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Preface 



For the thirty-second year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the national AECT Convention in Anaheim, CA. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volume #1 is 
available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes are 
available to members at AECT.ORG. 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Abstract 

The purpose of this study was to document the processes involved in two phases of formative evaluation of 
an adaptive web based system which individualized instruction such as content, interfaces, instructional strategies, 
and resources. The data were obtained through questionnaires, surveys, focus group interviews, and system logs 
throughout 2009. A total of 186 undergraduate students participated in two stages of this study. 

Introduction 

The major challenge and aim of Web-based instruction has been, and continues to be, to accommodate 
students with differing profdes, expectations, prior experiences, and learning abilities (Abidi, 2009; Azevedo, Moos, 
Greene, Winters, & Cromley, 2008; Dogan, 2008; Sikora & Carroll, 2003). Adaptive Web-Based Learning 
Environments (AWBLEs) are a form of adaptive instruction which attempt to address the issue of individual 
differences. AWBLEs do so by providing mechanisms for individualizing instruction (e.g., content, interface, and 
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strategies) and providing users with a more personal experience through the incorporation of differing instructional 
strategies, resources, assessments, and interfaces (Inan & Grant, 2008). Basically, AWBLEs gather user information 
and preferences (Brusilovsky, 2001; Triantafillou, Pomportsis, & Demetriadis, 2003; Tsianos et al., 2009); make 
inferences based on the collected data; and then employ various adaptive methods to accommodate each individual 
student (Far & Flashimoto, 2000; Dogan, 2008; Inan & Grant, 2008). 

Although the idea of what AWBLEs are and what they do is simple, the design, development, 
implementation, and evaluation processes of AWBLEs, however, is a challenging and complex topic. Moreover, 
although several adaptive web based systems have been developed, only a limited number of empirical studies have 
documented proof of the effectiveness of these systems on students’ performance, learning time, motivation, and 
attitudes. Therefore, it is crucial that more AWBLE designers document the formative and summative evaluations of 
their systems throughout the design and development phases. In addition to building on the literature, one of the 
major advantages of formative evaluation is that it will help researchers determine whether their system’s current 
design allows the system to meet its intended goals and objectives. 

Purpose of Study 

This study documents the formative evaluation (field testing) of an adaptive web based learning 
environment. The goals for formative evaluation were, first and foremost, to assess whether the goals of the system 
were being met (i.e. knowledge gains and motivation gains) and, secondly, to identify weak or problematic areas, in 
terms of usability, where the system could be improved. 

Specific research questions included: 

1) Flow do the students perceive the adaptive system? 

a) What were students’ perceptions on the adaptive tutorial in terms of visual design, 
organization/navigation, content presentation, use of multimedia, assessment feedback, and task value 
of the system? 

b) What did students like most about the adaptive tutorial? What did they like least? 

c) Did the students experience any difficulties with the adaptive tutorial? 

2) Flow does student learning, engagement, and satisfaction vary with the adaptive tutorial based on student prior 
knowledge and motivation? 

a) Based on knowledge or motivation levels (high or low), which groups of students benefited most in 
terms of knowledge acquisition? 

b) Based on knowledge or motivation levels (high or low), which group took the longest time to complete 
the tutorial? Which group took the least amount of time? 

c) Based on knowledge or motivation levels (high or low), which group rated higher satisfaction with the 
system? 



Methods 



Evaluation Design 

For field testing purposes, the researchers of this study used a tutor alone evaluation design (Woolf, 2009). 
In this evaluation design, no control group is used. Rather, only one group of students works with the same system 
and the study measures specific outcomes from that group. The goal of this evaluation design is to establish or 
identify something about the learner or the system which can be used to predict those outcomes on posttests (Woolf, 
2009). For this study, the outcomes of interest were student motivation and knowledge levels. Using data obtained 
from the adaptive system, researchers were able to account for prior motivation and knowledge levels in addition to 
measures of these two factors in posttests. 

Field-testing of the AWBLE included two stages. In the first stage, data were collected via the Internet. 
Participants linked to the AWBLE and studied online. During this stage, students’ prior and post knowledge, prior 
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and post motivation and their thoughts about the system were assessed. In the second stage, student studied the 
material in a computer laboratory individually. Participants were observed by the researchers to identify any 
technical problems that users faced. Participants of this stage were asked to fill out an evaluation survey. In addition, 
focus group interviews were conducted with these participants. 

Adaptive System Prototype 

Combining adaptive hypermedia methods with strategies proposed by instructional theory and motivation 
models, the adaptive online tutorial was developed in the fall 2008 and first half of the spring semesters. At the 
beginning of the tutorial, students filled out a pretest on their prior knowledge, and a survey of motivation. 
Depending on the results of these assessments, students were placed into one of four clusters for each of the three 
sections of the tutorial. Clusters were (1) low motivation and low prior knowledge, (2) low motivation and high 
prior knowledge, (3) high motivation and low prior knowledge, (4) high motivation and high prior knowledge. 

Based on gathered data, the adaptive tutorial automatically incorporated relevant adaptive strategies customized to 
student interest and needs. After which, students would continue through the adaptive content. Following the 
completion of each section, student motivation and achievement was re-assessed before proceeding into the next 
section. 

Participants 

The target audience for the system was undergraduate students. For field testing purposes, a total of 186 
undergraduates from a large southwestern university participated in two stages. Participants came from six sections 
of an undergraduate introductory technology course. More than half of the students participating in the initial stage 
were female (53.3%). Their ages ranged from 18 to 30 years old with a mean age of 19.8. The majority of students 
participating in the second stage were female (55.6%), between the ages of 17 and 26. 

Data Collection & Instruments 

• Achievement was measured using a locally developed 20 item multiple choice instrument over the introductory 
statistics topics covered in the tutorial. 

• Items adopted from the Instructional Materials Motivational Scale (IMMS) were used to measure student 
motivation level (Keller, 1987). Cronbach’s alpha for the IMMS ranged from .61 to .81 (Keller, 1987). 

• A Formative Evaluation Survey was used to gather information about of students’ use of the adaptive system 
and their perceptions on different utilities provided within the system. The survey had 44 items rated on a 
Likert-type scale of 1 to 5. 

• A Student Questionnaire was used to collect student perceptions about the adaptive system. Three open ended 
items asked students to elaborate on what they liked and disliked about the system and gathered their 
suggestions for system improvement. 

• An Observation Form was developed to help the designer take notes during the observation. System sections, 
observed problems and suggestions were written down on the form. 

• A Student Navigation Log was used to track student navigation patterns through the adaptive tutorial. This log 
recorded student actions such as usage time and the sequence of navigation through the tutorial. 

• An Interview Guide was primarily used to list questions and outline the topic to be investigated. Semi- 
structured focus group interviews were conducted in order to investigate students’ perceptions about adaptive 
system in more detail. Each interview lasted about 20 to 30 minutes and all interviews were audio recorded then 
transcribed. 
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Summary of Results 



Students Perceptions about Adaptive System 

The following table summarizes the evaluation of student perceptions. 



Criteria 


Findings 


Data Sources 


Implications 


Visual Design 


- Easy to read, easy to follow, not 
many words on the screen 

- Use highlight text to emphasize 
(color/underline etc) 

- Limit fancy fonts, flashy things 
may be distracting 


- Interviews 

- Questionnaire 

- SME Evaluation 


Minor updates to increase 
visual appeal and limit 
distractions 


Organization / 
Navigation 


- Navigation was easy and simple 


- Observations 

- Interviews 

- Questionnaire 


N/A 


Content 

Presentation 


- The practice examples creates 
interactivity making user feel that 
they are actively involved in the 
tutorial 

- The information may be easy, 
repetitive for those who known the 
topic 


- Interviews 

- Questionnaire 


Revise prior knowledge 
assessment to provide 
better adaptation for 
student with high prior 
knowledge 


Assessment & 
Feedback 


Some of the questions were hard 
and/or wording from the 
questions was confusing 


- SME Evaluation 

- Questionnaire 


Improvement on the 
knowledge test 


Motivation 


Probability is considered as a 
boring subject or irrelevant to 
student major 

Motivation survey questions were 
a little repetitive 


- Questionnaire 

- Interviews 


Field test in the setting 
where student hold 
students accountable for 
learning the instructional 
material (Stage II) 

Revise motivation survey 


Technical Issues 


Browser crashed a few times, 
while students using practice 
activities 


- Observations 


Re-test practice activities 
using different version of 
Flash Player 



Student Learning, Study Time, and Satisfaction 

Both high and low motivation groups benefited from the system and showed an increase in their knowledge 
levels. There were no significant differences between the low and high motivation groups’ satisfaction ratings. 

Based on the descriptive statistics, the students with high motivation spent more time studying the tutorial than 
students with low motivation. 

Results indicated that high knowledge groups rated significantly more satisfaction with the system than low 
knowledge group. Based on their prior knowledge levels, high knowledge students spent more time studying the 
tutorial than the low knowledge students. 

Discussions 

Based on data obtained from the formative evaluation processes, researchers iteratively updated the initial 
prototypes by increasing visual appeal and limiting distractions; revising prior knowledge assessments to provide 
better adaptation; notifying students about the number of sections and the anticipated completion time on the login 
page; and revising the wording on certain topics in order to accommodate students of low prior knowledge. For 
students with high motivation and high prior knowledge, designers have considered adding more content which may 
be more challenging and engaging for these students. Further, designers have considered adding additional 
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navigation links which would allow these users to access additional topics. In order to increase relevance, designers 
have been working on including more examples and practices which relate probability to student interests or majors. 
In addition, designers plan to redesign examples and practices to be more appropriate for college-age students. It is 
anticipated that these new features will increase student motivation by showing relevance. 
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Abstract 

The goal of this study was to investigate the 5 th grade students’ attitudes toward animated 
pedagogical agents (APA) in language learning environment. 187 5 th grade students from two different 
regions of Turkey participated in the study. Data were collected through a four-point Likert type attitude 
scale developed by the researcher. The results indicated that the participants interpreted the APA as a 
helpful partner having human-like behaviors. Furthermore, APA supported and facilitated their learning. 

Theoretical Framework 

As technology advances, the demand for applying technological tools in the education for various 
purposes increases. One of the latest technological evolutions is animated pedagogical agents (APA) which 
are likable computerized characters with embodied life-like behaviors such as gesture, gaze, locomotion, 
speech emotions, etc. (Moreno, Mayer, Spires & Lester., 2001). Because these non-verbal and verbal 
behaviors also occur in human- human interaction, APAs having these behaviors add powerful capability to 
multimedia learning environments (Atkinson, Mayer & Merill, 2005). These behaviors also let the people 
to perceive the environment as a social context (Moreno, et al., 2001; Dunsworth & Atkinson, 2007). Thus, 
the naturalness of the human computer interaction can be maximized by the use of APAs in the multimedia 
learning environments. 

The pedagogical agents’ effectiveness in the human computer interaction is supported by social 
agency theory. According to this theory, using verbal and visual cues in the computer-based environments 
encourage the learners to interpret the computer as a partnership. They consider their interaction with the 
computer as social one, because the social cues are similar what they would expect form a human-to-human 
conversation (Atkinson, et al., 2005). 
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Recent research studies in the multimedia learning environments and human computer interaction 
has begun to focus on the educational benefits of animated pedagogical agents. The effectiveness of APA 
in educational setting has been studied in different fields, such as science, mathematics and humanities 
(Atkinson, 2002; Baylor & Ryu, 2003; Moreno, et ah, 2001). However, limited research has investigated 
the use of pedagogical agents in second/foreign language learning. Even though various external and 
internal properties of the APA various can be beneficial in language learning, the application of APA in 
multimedia learning environment for second language learning has limitedly investigated. In addition, the 
recent research studies generally cover the undergraduate or adult learner. This study examined the 5th 
grade students’ attitudes toward animated pedagogical agents in second language acquisition. 

Method 

The participants of this study were 187 fifth grade students (99 females, 88 males) from two 
different regions of Turkey. Participants were between ages of 10 and 11, (MM 0.91, SD=.51). 

Multimedia learning tool was developed by the researcher by considering the needs of the course teacher, 
the content of the lesson and levels of the students. This multimedia learning environment aims to teach 
and practice the model verb “can” to the 5th grade students in their English as a second language courses. 
The “Peedy”, Microsoft animated agent was used in these learning environments. After studying the 
multimedia learning tool, data were collected through four-point Likert type attitudes toward animated 
pedagogical agents scale developed by the researcher. Some items were written according to the social 
agency theory and some items were adopted from existing instillments used for assessing the learners’ 
attitudes toward animated pedagogical agents (Ryu & Baylor, 2005; Adcock & Van Eck, 2005). Reliability 
coefficients for four factors (supporting to learn, facilitating learning, human-like and being a partner) of 
this instrument are .77, .82, .68 and .70 respectively. 

Results and Discussion 



Supporting to learn 

Participants’ attitudes toward the Peedy in term of supporting to learn was high with a mean of 
3.32 and standard deviation of .62 on a four-point likert scale. This indicated that APA is a beneficial 
technological tool to support the young students’ second language learning. He supported the students’ 
learning by giving feedback and clues and taking their attention to important points. 
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Facilitating to learn 



Participants perceived the Peedy helpful in term of facilitating their learning with a mean of 3.41 
and standard deviation of .58. They found the hand movements of the Peedy helpful for concentrating on 
the important concepts. Peedy made the content enjoyable, entertaining and interesting. Furthermore, Peedy 
guided the participants in the use of multimedia tool and presents the content effectively. 

Fluman-like 

Participants perceived that Peedy has life-like behaviors and characteristics (M = 3.08, SD= .68 ). 
They thought that Peedy’s hand movements, gestures, gaze, locomotion and speech emotions were similar 
to the people. 

Being a partner 

Participants interpreted the Peedy as a social partner in multimedia learning environment 
characteristics ( M = 3.32, SD= .59 ). They thought that they were a good team together and get along with 
well like friends. 

Conclusion and Discussion 

APA, which provides instruction through verbal and nonverbal modes of communication, yields a 
social interaction between the computer and learner. Recent research studies investigated the different 
roles and aspects of pedagogical agents on the retention, transfer, interest and persona affect. Some of the 
pedagogical agents are Flerman the Bug (Moreno et. al, 2001; Lester, Converse, Kahler, Barlow, Stone & 
Bhogal, 1997), and Steve (Johnson, Rickel &Lester, 2000). These agents have a role of tutor, motivator, 
expert, mentor and learning companion. These studies concluded that animated pedagogical agents enhance 
the students learning, motivation and their interest to the learning environment. Paralell to these studies, the 
results of this study indicated that 5 th grade students perceived the APA as a social partner who shows 
human-like characteristics. APA is also a beneficial to support and facilitate learning especially in second 
language acquisition. 

Ohmaye (1998) stressed the importance of interaction in language acquisition. He mentioned that 
“language learning depends heavily on interactions with native speakers, and native speakers are hard to 
come by” (p.2) He continued explaining the limitations of the classroom instruction in the interaction with 
the native speakers, access target culture and individual attention and feedback. However, Turkish students 
do not have enough opportunity to interact with native speakers because of limited resources, social norms 
and personality traits. However, multimedia learning environments with APA offer students opportunities 
to interact with native speakers and provide a social context (Ohmaye, 1998). Therefore, this study 




exemplifies the use of APA in second languge learning and provides the attitudes of the young learners 
toward APA. 
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Innovative Strategies for the Redesign of Asynchronous Field Trips by 
Employing Cyber Technologies and GPS Devices: a Situated Learning for 

Hands-on Experience 

Xin Chen 
Hongxia Yang 
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Abstract 

This design-based research presents an innovative approach for a field trip, which employs cyber learning tools such 
as Google Maps, ePortfolio, university course website, and GPS devices, to offer students an environment of 
situated learning. This paper details the possible solutions and advantages of students self-guided field trips to 
overcome the challenges existing in a traditional field trip, such as the problems of passive learning, as well as 
resource and time consumption on field trip planning and implementation. 

Key words: field trip, GPS, situated learning 



Problem Statement 

Field trip, as an important instructional means, significantly impacts learning in the cognitive and social 
aspects, affording learners great opportunities to have a concrete understanding on subjects and to obtain hands-on 
experience (Tal & Morag, 2009). Although field trips have been widely used in K-12 and higher education, there are 
still a variety of challenges in its design, management and implementation. A common problem of the field trip is 
the simple transformation from a classroom lecture to an instructor-led outdoor activity, which merely adds a “look 
and see” experience (Beierdofer & Davis, 1994). This type of activity deviates from the principles of “situated 
learning”, which is to encourage learners to take an active role in learning, allow them to construct knowledge 
instead of receiving information, as well as afford them a situation to comprehend and apply knowledge instead of 
simply repeating what was learned (Marlowe & Page, 1998). Furthermore, the field trip design, management and 
implementation is an intensive process. Many field trips can hardly meet requirements, such as: (1) instructors’ 
training in group management skills, (2) instructor/student ratio no higher than 1:10, (3) pre-trip planning the 
resources, (4) schedule coordination, and (5) suitable weather conditions (Beierdofer & Davis, 1994). Many field 
trips fail because of these challenges (Collins & Redcross, 2005; Floff, 1942; Martin & Seevers, 2003). One such 
example is a biology class at an east coast university. Currently, each instructor takes 24 students on a campus tour 
to visit plants. The initial increase in the students’ motivation by employing a field trip fades quickly after visiting 
the first two or three trees. This is due to the fact that receiving a lecture in a field is no better than that in a 
classroom. During the field trip, the instructor demonstrates plant structures by using a real leaf, seed or fruit. 
Students standing far away from the instructor can barely see the examples. More seriously, during the field trip, 
students simply follow the instruction and passively receive information. They have little chance to observe, touch 
or feel the plants by themselves. This is not consistent with the original intention of the field trip: allowing learners 
to have hand-on experience. A desired field trip should afford students an effective and enjoyable first-hand 
experience in which they are intentionally engaged in authentic activities allowing observation, elaboration and 
knowledge construction (Carroll, 2007; Lei, 2010; Rudmann, 1994; Peterman, 2008). In addition, the application of 
innovative technology could alleviate some of the challenges often seen in the field trips. 

Description of Study 

This design-based research project addresses the field trip challenges mentioned above. The proposed 
project converts an instructor-led field trip into an asynchronous, students self-guided field trip. Learners will form 
teams to take a trip on campus, looking for plants covered in this lesson. To arouse students’ interests, the element of 
Geocaching (a treasure-hunting adventure) is added in the field trip. Each plant serves as a geocache waiting for the 
students to discover. Since this self-guided tour will be conducted on the university campus, safety issues won’t be 
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a concern. Google Maps containing the geocache locations will be provided. Students will also use GPS devices to 
guide their journeys. All the guidelines and instructional materials will be available on the course website. Students 
are asked to print out the handouts before the field trip. Once students find a plant, they will read the information 
about this plant on the handout. They will also observe it, take notes of it, take pictures, and answer the questions on 
the handout. Upon the completion of the tour, each team will create an ePortfolio to reflect what they have learned. 
An ePortfolio encourages learners to take the responsibilities in their knowledge construction, supports peer-to-peer 
review, facilitates the process of knowledge sharing, as well as serves as evidence of effective learning assessment 
(Reeves & Okey, 1996). 



Methodology 

This project focuses on the design and implementation of an asynchronous, students self-guided field trip at 
the university level. The researchers will randomly select 5 out of 20 classes as experimental groups. The total 
number of samples will be five instructors and around 120 students. The study will be implemented for 1 week 
during Spring 2011. After the first implementation, the researchers will conduct one-on-one interviews with the five 
instructors to find out the coherent themes on what works well and what does not work in this field trip. A survey 
will also be used to collect data about students’ opinions on the field trip. The survey will include some close-ended 
(Likert scale) questions and a few open-ended questions to investigate the factors of class effectiveness from the 
students’ perspectives. The researchers will look for the correlation between the students’ satisfaction and the 
learning outcomes, as well as conduct the regression analysis using the students’ satisfaction and the learning 
outcomes to predict the effectiveness of the field trip. 

Application of Results 

Design-based research supports design progress as an iterative process where research, design and 
pedagogical practice are tightly intertwined (Joseph, 2004). Therefore, the research team will revise the field trip 
design based on the result of the first round data analysis. After the revision, it will be re-implemented in Fall 201 1, 
and then a second round data will be collected for further refinement. This research intends to improve the 
effectiveness of the field trip, increase the resources economization on the special training of instructors, as well as 
to save efforts on pre-trip preparation and schedule coordination. The format of this field trip is logistically feasible 
to be adopted for other natural science courses. Any persons involved in the design, development, implementation, 
and/or administration aspects of authentic learning experiences will benefit from this project. This research will be 
of interests to the instructional design and technology professionals in K-12 and higher education. 
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Abstract 

This study investigated the reversed modality principle in spatial learning content with two different 
modality conditions. Participants were randomly assigned to two groups (visual text and spoken text). The findings 
revealed no significant differences in terms of mental effort for the instruction and assessments, the usability level, 
and perceived usefulness. However, the significant effects on three assessments showed that the visual text group 
performed better than the spoken text group. The results support a reverse modality effect. This study provides 
theoretical support for establishing boundaries for the modality principle as well as practical implication for 
instructional designers. 



Theoretical Framework and Hypothesis 

The modality principle in multimedia learning has received considerable research support over the last few 
decades (Ginns, 2005; Mayer, 2001; Sweller, van Merrienboer, & Paas, 1998). This principle states that learning 
from words and pictures is improved when written or on-screen text is replaced with spoken text. A theoretical 
rationale for this principle is provided by Baddeley’s (1992) model of working memory. According to Baddeley, 
working memory contains two sub-systems, one for processing visual information and another for processing verbal 
information. Presenting textual information visually (as on-screen text) during multimedia learning is purported to 
overload the visual subsystem and strain attentional resources. This occurs because of the need to temporarily hold 
and process text along with pictorial information (e.g., animation) in the same memory subsystem (Mayer & 
Moreno, 1998). However, according to the modality principle, this unimodal presentation format can be improved 
by employing a bimodal format, wherein textual information is presented auditorily and pictorial information is 
presented visually. This presentation format is purported to reduce cognitive load by using the total capacity of 
working memory (both visual and verbal subsystems) more efficiently (Tabbers, Martens, & van Merrienboer, 
2004). The modality principle has been linked to reduced mental effort and study time during instruction and to 
improved performance on retention, transfer, and matching tests (Tabbers et al., 2004). 

The modality principle has been linked to reduce mental effort and study time during instruction and to 
improved performance on retention, transfer, and matching tests (Ginns, 2005; Tabbers et ah, 2004). The modality 
principle has also been validated across a variety of computer-based media, such as multimedia explanations, agent 
based computer games, and virtual reality (Moreno, 2006). 

Notwithstanding the research support for the modality principle, recent research has shown that the 
principle may not apply to all multimedia learning situations. For example, Tabbers et ah (2004) found that a visual 
presentation of text was superior to a spoken presentation when learners were given control over the pacing of the 
instruction. There is also evidence that visual text may be superior to spoken text if the subject matter pertains to 
learning spatial relations (Penny, 1989). 

In the current study we examined the generalizability of the modality principle to an instructional situation 
that prior research suggests may be conducive to a reverse modality effect. That is, where on-screen text is superior 
to spoken text. To create this situation the following elements were incorporated into the treatment materials: 1) 
learner control of instructional content, and 2) a learning task with a significant spatial component. Our hypothesis 
was that learners studying on-screen text would outperform those studying spoken text due to the presence of 
learning conditions favorable to a reverse modality effect in multimedia learning. 
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Method 



This study investigated the effects of modality on spatial knowledge learning from a computer-based 
diagram and related text. Modality (visual text vs. spoken text), see Figure 1, were examined between two groups. 
One hundred and seventeen undergraduate students from a large southwestern university were randomly assigned to 
one of the two experimental conditions. 



Modality 




Figure 1. Screen shots from the visual (left) and spoken text conditions 



The experimental materials consisted of a computer-presented diagram depicting 12 places of articulation 
in human speech. The participants accessed the text by selecting a place of articulation on the diagram with a 
computer mouse. In the visual text condition, descriptive sentences were presented as on-screen text; in the spoken- 
text condition the text was presented as narration. The participants had complete control over the pace and sequence 
with which they selected the hyperlinked text. They were instructed to study the diagram and the related text for 10 
minutes. After 10 minutes of study, the participants were given the survey, reconstruction, labeling, and matching 
assessments. 

The dependant measures consisted of the number of clicks on description dots, the number of clicks on 
replay buttons, mental effort, the usability level, the usefulness of either visual or spoken text, a diagram 
reconstruction test, a special labeling test, and a matching test. All dependant measures were delivered via computer. 
In the instruction, there are 12 different round markers in the diagram as shown in Figure 1. Once they click a 
marker, they can see the written text or listen to the spoken text in a pop-up box. Both types of text will stay for 25 
seconds, and a review or replay button will be shown after the written text disappears or the spoken text finishes, 

See Figure 2. The two different numbers of clicks tracked how many times participants revisit or replay the 
descriptions for each place. 
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Figure 2. Review and replay screens 

Subjective mental effort ratings were obtained after the experimental treatment and after each of the three 
assessments. These ratings measured (on a 7-point Likert scale) the amount of cognitive resources the participants’ 
perceived they invested in the instruction and the three assessments. This subjective measurement scale has been use 
to calculate instructional efficiency scores in many studies investigating cognitive load theory (e.g., Kalyuga & 
Sweller, 2005; Tuovinen & Paas, 2004). The level of usability measured how easily participants operated or 
navigated the interface. The usefulness of text types represented participants’ perceptions toward either visual text or 
spoken text. The reconstruction test presented each participant with a blank outline of the Places of Articulation 
diagram and a listing of the 12 places of articulation next to the diagram. Each participant was required to drag the 
name of each place to its correct location on the diagram. We measured the distance between correct locations and 
the locations participants moved. Similar to the reconstruction test, the spatial labeling test provide the unlabeled 
places in the articulation diagram and the list so that participants can move a description to the right place. This test 
measured only right or wrong places. Finally, the matching test consisted of a listing of the 12 places of articulation 
(e.g. Dental) and a corresponding listing of the sounds made at each place (e.g., “th” as in thunder), plus three 
distracters. The participants were required to match each place with its corresponding sound. 

Results and Conclusions 

Independent /-Tests were conducted to compare all variables between groups. The results revealed that 
there were no significant differences in terms of mental effort, usability, and usefulness. However, five dependant 
variables had significant differences. Regarding the numbers of clicks, the numbers of clicks for the descriptions (M 
= 36.80,/? < .001) and for the replay button (M = 1.39,/? = .047) in the visual text group than the numbers of clicks 
(M = 17.39 for the description, M = .82 for the replay) in spoken text group. In addition, the visual text group had 
higher scores in all test scores as shown in Table 1. Note that the values of the reconstruction test represent the 
distance from the correct place, so lower numbers mean higher scores. In summary, we observed a reverse modality 
effect for all dependant measures, as hypothesized. 

Table 1. Results of test scores 



Test 


Mean scores 


Independent t-test 




Visual Text 


Spoken Text 




Reconstruction 


66.14 (51.75) 


90.61 (52.72) 


?(1 15) = -2.533,/? = 013 


Spatial labeling 


8.87 (2.96) 


7.57 (3.42) 


?(1 15) = 2.197,/? =.030 


Matching 


3.54 (2.59) 


2.60 (1.66) 


t(\ 15) = 2.294,/? = 024 



The results of this study support the results of the study by Crooks et al. (2009) and Tabbers et al. (2004). 
Modality effects are not likely to occur with diagrams and text when learners have control over the pacing and 
sequence of their study and spatial learning is an important instructional outcome. In fact, we observed a reverse 
modality effect favoring visual text over spoken text. For example, the visual text group had more control over 
pacing while the spoken text group had to stay for 25 seconds to listen to whole narrations. So, the visual text group 
had more chance to revisit and review the description. The best explanation appears to be that user control and 
spatial learning are conditions favorable to learning from on-screen text. These results will guide practitioners 
during the design and development of multimedia materials and to theorists as they seek to clarify the boundaries of 
the modality effect in multimedia learning. 
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Abstract 

This study investigated the effectiveness of organizers in a serious game as an instructional unit. For this 
study, a serious game about transistor recycling was used as a tool for game-based learning. The results revealed that 
a post organizer provided, after the game was finished, positively influenced the participants’ recall of the learning 
content and their perceptions of enjoyment, learning tool, and satisfaction toward the game. However, no organizer 
and an advance organizer had no effect. 



Introduction 

Computer games have been applied to learning environments as an effective instructional tool (Dormann & 
Biddle, 2006; Gee, 2003; Mansour & El-Said, 2008). The educational utilization of computer games has been 
defined as game-based learning (Squire, 2008). Specifically, Gee (2003) argued that game -based learning is a virtual 
learning space where learners enable to experience such unique attributes in computer games as interactivity, 
enjoyment, and engagement while they attain learning goals in traditional or online learning environments. For 
instance, instructional software “Computer Tutors,” which was developed by BBC, provides a game-based learning 
course utilizing gaming strategies. Thus, game-based learning refers to the strategic application of computer game 
for learning. 

Serious games have emerged as a pragmatic unit for game -based learning, which is initially designed with a 
primary purpose rather than for entertainment or enjoyment (Freitas, 2006; Squire, 2008). Therefore, serious games 
can be considered an attempt to specifically apply the computer game attributes, such as enjoyment and engagement, 
to learning environments (Koster, 2005; Kirkley & Kirkley, 2005). It has been found that serious games stimulate 
students’ learning motivation, subsequently leading to the increased rate of knowledge acquisition and retention (e.g., 
Aldrich, 2005; Michael & Chen, 2006), such as “Immune Attack” which was developed for the purpose of educating 
students about basic concepts of cellular and molecular biology. However, considering that serious games should be 
situated between the activities of entertainment and pedagogy (Dormann & Biddle, 2006), it is challenging to utilize 
serious games as a learning tool (Kirkley, Tomblin, & Kirkley, 2005; Mayo, Singer & Kusumoto, 2005). Therefore, 
there is a need for more research that examines the effective use of serious games. 

Previous studies have examined the effect of organizers as supportive materials to learning. It has been 
suggested that advance or post organizers can enhance students’ comprehension in traditional learning environments 
by increasing their attention to important information (Butz, Miller, & Butz, 2005; Lucas & Goerss, 2007; Robinson, 
Katayama, Beth, Odom, Hsieh, & Vanderveen, 2006). Therefore, we expect organizers to effectively aid the 
educational use of serious games. Currently, most of the existing serious games contain brief information, such as 
how to play the game; however, there have been few games containing organizers that effectively describe the 
content of the game. Therefore, the purpose of this study is to examine the effects of organizers in a serious game. 
This study, specifically, determines which organizers would be more effective between advance and post organizers. 

Research Questions 

The research questions for this study are: 

1. Do organizers in serious games influence students’ game scores? 

2. Do organizers in serious games influence students’ recall of the learning content? 

3. Do organizers in serious games influence students’ perceptions occurred by playing the game? 

Method 

Participants for this study were 99 undergraduate students (mean age = 20 years old) at a southwestern 
university. This study had three conditions: (a) group 1 - no organizer (N= 35), (b) group 2 - advance organizer ( N 
= 35), and (c) group 3 - post organizer (N= 29). 
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This study, consisting of three phases, was carried out in a computer lab. In the first phase, the participants 
were asked to play a serious game ‘ The Transistor ,’ which was developed by Nobelprize.org (see Figure 1), for 15 
minutes. Transistor and non-transistor items were featured throughout the game in order to teach the student the 
importance of transistor recycling. In the game, they were asked to remove all non-transistor items that do not have 
transistors by using a mouse click. An additional Web page, showing a list of the items appearing in the game, was 
embedded in the game. It functions as an organizer intending to aid students to better understand transistor and non- 
transistor items. When the organizer page appeared before playing the game, it was defined as an advance organizer. 
When the organizer page appeared after playing the game, it was defined as a post organizer. In the second phase, 
they were asked to answer questions about their perceptions of the game they previously played. Their perceptions 
of enjoyment, learning tool, and satisfaction were measured using a 7-point Likert scale ranging from ‘strongly 
disagree’ to ‘strongly agree.’ In the third phase, they were asked to answer questions assessing their recall of 
transistor and non-transistor items appearing in the game. They were asked to list the items correctly recalled among 
a total of 21 items. 




oooo 



Transistor Recycler Game 



Thb Transistor 



ooo 



You will have 15 minutes to play this game. Please remember the highest level and score you have got. 
Following the instructor’s direction after 15 minutes, select "next" button. 



Next|B|r 



Figure 1. A screenshot of the serious game ‘ The Transistor’ 

Results 

A one-way ANOVA was computed to determine the differences of game scores between groups. The result 
showed that there was not a significant difference in terms of the location of organizers (F( 2, 96) = 1.68, p = .193). 
The mean scores of group 1, group 2, and group 3 were 121.66 ( SD = 37.80), 138.89 (SO = 44.04), and 130.41 (SO 
= 35.00), respectively. 

Recall of learning content was analyzed by performing a one-way ANOVA. The analysis revealed that 
there was a significant difference in terms of the location of organizers, (F( 2, 96) = 4.07, p = .020). A Tukey post- 
hoc test showed that group 3 (M= 12.24, SD = 2.98) had better recall than group 2 ( M= 10.40, SD = 2.59). 
However, there was no difference between group 1 (M = 10.80, SD = 2.47) and group 3, and between group 1 and 
group 2. 
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Another one-way ANOVA was conducted on the perceptions toward the serious game. These perceptions 
were enjoyment, learning tool, and satisfaction. First, the analysis revealed that perceived enjoyment was 
significantly different in terms of the location of organizers, (F( 2, 96) = 3.13, p = .048). A Tukey post-hoc test 
suggests that group 3 (M = 4.72, SD = 1.62) had a higher level of perceived enjoyment than group 1 (M= 3.74, SD = 
1.49). Flowever, there was no difference between group 1 and group 2 (M = 4.09, SD = 1.61), and between group 2 
and group 3. Second, it was found that perceived learning tool was significantly different in terms of the location of 
organizers, (F( 2, 96) = 3.51,/? = .034). A Tukey post-hoc test showed that group 3 (M= 4.99, SD = 1.58) had a 
higher level of perceived learning tool than group 1 (M = 3.99, SD = 1.70). Flowever, there was no difference 
between group 1 and group 2 (M = 4.16, SD = 1.46), and between group 2 and group 3. Third, it was also found that 
satisfaction toward the game was significantly different in terms of the location of organizers, (F( 2, 96) = 5.09, p 
= .008). A Tukey post-hoc test showed that group 3 (M = 4.69, SD = 1.60) had a higher level of satisfaction than 
group 1 (M= 3.46, SD = 1.52). Flowever, there was no difference between group 1 and group 2 (M= 4.18, SD = 
1.54), and between group 2 and group 3. 



Discussions and Conclusion 

Based on the results, the use of post organizers should be encouraged to enhance positive educational 
effects of serious games, such as recall of the learning content and perceptions of enjoyment, learning tool, and 
satisfaction. Post organizers were proven to help students develop a well-established schema for the learning content, 
ultimately causing better recall. An interesting aspect, found in this study, was that post organizers positively 
influenced the students’ perceptions toward the serious game. Thus, it can be assumed that the higher knowledge 
acquisition enhanced by post organizers yields not only higher recall scores but also positive perceptions toward the 
game. Consequently, post organizers, as an instructional component, could be recommended to strengthen the 
effectiveness of serious games. 

Flowever, the insertion of advance organizers was not shown to be effective on the students’ recall and 
perceptions. It can be assumed that advance organizers did not impact learning because the students tended to forget 
the earlier insertion of organizers during the game. Within the gaming environment, post organizers can be more 
effective than advance organizers. 

When considering the nature of the empirical research, this study revealed several methodological 
limitations that should be considered in interpreting the results discussed above. First of all, this study failed to show 
a significant difference in the game scores in terms of the location of organizers. However, this study asked the 
participants to report only the best score they received while playing the game repeatedly. Future research needs to 
examine the effects of organizers on the scores over time and examine each student’s score simultaneously. Second, 
the game used for this study is simple and some participants might bore easily. Thus, future research can use more 
serious or complex games to stimulate the students’ enjoyment. Third, the effects of organizers on students might be 
different in other kinds of serious games. In addition, the sample size of this study was somewhat small. These could 
prove to be critical issues that do not allow the results of this study to be generalized. Or, a larger sample size might 
be used to establish a generalization of the results. Last, a different type of organizers could be considered if a game 
focuses on affective domain and psychomotor domain rather than cognitive domain. 
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Abstract 

The purpose of this needs assessment was to determine student and program faculty concerns about 
scheduling courses, related to online and on campus delivery. The needs assessment was a two-part process that 
took place over two semesters. Extant data, questionnaires, interviews, and focus groups were used to address 
whether students were satisfied with their ability to schedule courses and identify any concerns related to being able 
to complete their degrees. Most respondents expected to complete their degrees in a timely manner and were 
satisfied with the instruction they received overall. Some stated they were not always able to take courses they 
preferred or when needed, and could take other available courses instead to keep making progress. Concerns about 
course delivery were expressed, but mixed; that is, some expressed the need for more online courses while others 
expressed a need for more on campus courses. 



Introduction 

Needs assessments are conducted for a variety of purposes, but an overarching purpose, in an academic 
setting, is to gather information from a variety of sources to determine if there is a gap between actual (what is) and 
optimal situations (what should be) (Rothwell & Kasanas, 2004, Altschuld & Kumar, 2010). In other words, it is 
about clarifying problems and providing data for decision making (Gupta, 2007). Unlike other inquiry methods (i.e., 
case studies, narratives, and so on), a needs assessment includes, not only the results, but also provides 
recommendations or solutions. Kauffman, Rojas, and Mayer (1993) suggested that stakeholders may also receive 
advice as to the consequences and payoffs if, or when, the need assessors’ recommendations are followed. 

One of the first tasks is to gather data from a variety of sources to gain a basic understanding of the 
stakeholders’ needs (Rothwell & Kasanas, 2004; Altschuld & Kumar, 2010). Based on information from this 
preliminary stage, the needs assessor assists stakeholders in determining whether all identified needs can be 
addressed or resolved, and, if necessary, prioritize them in terms of significance and positive impact on the 
organization (Moore-Thomas, C & Erford, B. T., 2004). Witkin and Altschuld (1995) suggest there are three levels 
of stakeholders. Within higher education institutions, level one, service receivers, are the students and level two, 
service providers, are primarily the faculty and staff. The third level is the resources and support structure that 
allows the second level to provide services (Altschuld & Kumar, 2010); in colleges and universities, this includes 
administrators and the infrastructure within the setting. 

Although organizations, such as universities, often make decisions with little to no consideration of 
conducting needs assessments (Watkins, 2005), instructional designers maintain that needs assessments provide 
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valuable information to meet stakeholders’ needs and organizational goals (Ingram, Haynes, Davidson-Shivers, & 
Irvin, 2005; Rothwell & Kazanas, 2004). Needs assessments also provide them with opportunities for better 
planning which, in turn, leads to actions for improving the instruction and learning. 

New academic programs 

Schuttler (2010) suggested that anytime universities consider developing new undergraduate or graduate 
programs, they should conduct a needs assessment. It could provide information about existing programs within 
their university as well as similar programs at other universities. The needs assessment could examine the 
availability and qualifications of existing faculty. In addressing the viability of adding new programs, the results 
could also establish practical and efficient access [for the potential student body to attend classes], estimate costs 
related to students obtaining degrees and costs in terms of time and resources to the university. For example, Hoyt, 
Howell, and Young (2009) conducted a needs assessment (NA) to determine the availability of evening courses for 
students. The purpose was to gather information about students’ needs relating to the availability of evening courses 
and their impact of scheduling on campus courses in the evening. This contributed to more efficient degree 
completion and satisfaction of nontraditional students and traditional students. They found that adding evening 
classes allowed more students to complete their degrees in a timely manner and, in turn, allowed the university to 
admit a new pool of students. 

Existing academic programs 

Needs assessments are conducted to provide information about existing programs as well (Davidson- 
Shivers, Inpornjivit, & Sellers, 2004; Ingram et al., 2005). Examination of degree programs may show areas of 
improvement in regards to the use of technology and how online delivery can be improved. (Martinez, Liu, Watson, 
& Bechelmeyer, 2006) 

Specific needs assessments in an instructional design program 

Three needs assessments (NAs) were conducted on the instructional design (ID) program in recent years at 
a regional university in the Southeast; each was focused on a specified purpose. The first NA, conducted in 2002, 
determined whether the ID program met students’ educational needs and enabled them to meet their professional 
goals (Davidson-Shivers, et al., 2003, 2004). Results indicted existing students and alumni expressed overall 
satisfaction with their education and that the ID program assisted in helping them meet their professional goals. 
Alumni also indicated that they had jobs in ID or a related field, which required ID knowledge and skills. The 
second NA, conducted three years later, assessed the program’s success in career preparation of students, but also 
determined alumni involvement in the program (Ingram, et al., 2005). The results verified the first NA findings and 
showed that alumni were interested maintaining a continued involvement with the program; alumni also provided 
recommendations of how it could be accomplished. Based on results of the first two NAs, suggested changes to the 
program were minor and some of the recommendations were implemented. Later, and as a course project, Clinton 
and Stevenson (2007) conducted a NA that focused on student retention and recruitment. Their findings indicated 
that student enrollment had been in decline for a few years and determined a need for increased recruitment effort; 
suggestions on ways to recruit new students were included in their report. The faculty took the findings under 
advisement and implemented some suggestions based on available resources and support structures in place. 

Purpose of Current Needs Assessment 

With this decline in student enrollments and a major effort by the university toward moving to more online 
course delivery, some concern about meeting student needs were being raised by both students and faculty. These 
voiced opinions indicated there may be a problem with course availability, scheduling, and form of delivery. At the 
same time, faculty were in the process of identifying an articulated curriculum for required and support course 
offerings; this curriculum was to reduce offerings of the same course per year in order to increase class size per. 
Hence, the needs assessment had the potential to verify expressed concerns as well as help with the sequencing of 
courses within the ID curriculum. 

The purpose of this current needs assessment was focused on identifying the current state of the ID 
program as related to course availability, scheduling, and delivery. Specifically, it was conducted to address whether 
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students were able to schedule courses satisfactorily and complete their degrees in a timely manner as well as gather 
information as to whether online or oncampus course delivery impacted student satisfaction levels. 

The needs assessment was conducted in two parts. The first part, the preassessment and assessment phase, 
was conducted as a Needs Assessment course project in summer 2009. The focus of this Part One was to verify 
voiced concerns using extant data about the program and the previous needs assessments’ findings and, then, to 
determine a need existed and, then, gather additional data from students and faculty as needed. Part Two, the follow- 
on assessment, was added for further exploration about findings from Part One. 



Methodology 



Organization, stakeholders, and team 

Organization. This needs assessment was conducted on an Instructional Design (ID) program at the 
University of South Alabama, in Mobile, AL. There are three degree programs: master’s, doctoral foundation track, 
and doctoral. The master’s program is a minimum of 40 credit hours and students complete this degree entirely 
online or oncampus, or with a mix of online and on campus courses. The doctoral foundation track program, which 
began in the academic year of 2005-2006, is for students with a bachelor’s degree or with a master’s degree in fields 
other than ID. Students must successfully complete 22 credit hours of master’s foundational core courses and pass a 
comprehensive exam on the foundational core to enter the doctoral program. Once enrolled in the doctoral program, 
continuous enrollment is required. The doctoral program is a year round program and requires a minimum of 60 
hours that include successful completion of doctoral core and support courses, research and statistic courses and 
research/dissertation course hours. Doctoral students must also be successful in completing a research and statistics 
exam, doctoral comprehensive exam and a dissertation study (University of South Alabama Graduate Bulletin, 
2010 ). 

ID courses, especially master’s and doctoral core courses (i.e., content directly related to the ID standards 
and competencies) were offered two times a year, with one delivered on campus and the other online. Research and 
statistics courses were generally offered at least once a year depending on the ID program and other college 
programs’ enrollments and needs. Support courses were offered once a year or maybe every other year, again, 
depending on student need. 

During summer term, ID faculty generally focused on core courses within the ID program or service 
courses required of all College of Education (COE) master’s students were required take (i.e., an introductory course 
on learning psychology and introductory course to measurement and evaluation). A few ID support courses were 
offered during the summer as well. 

Stakeholders 

Generally speaking stakeholders within university settings include students, faculty, administration, alumni 
and the university as a whole. The primary stakeholders for this project were the students enrolled in one of the three 
programs and the program faculty. 

Students. Seventy four graduate students were enrolled in one of the three programs at the time of the 
study. As noted by the enrollment records at the time of the NA, there were 17 students in the master’s program, 12 
doctoral track students, and 45 doctoral students. 

Faculty. Although we primarily focused on the needs of the students, we also considered the needs and 
concerns of faculty. The ID program has seven full-time faculty members. Five of them have doctorates in 
instructional design and the other two have doctorates in measurement and statistics or educational psychology. 

Needs assessment teams 

Part One: Preassessment and assessment phase. For Part One, students (/? = 5) enrolled in the Needs 
Assessment course, were assigned to two teams. Two were in the master’s program with the remaining three in the 
doctoral program. The two master’s students were placed on different teams. One team had two members, both 
female students; the other team had three members, one female and two male students. These teams worked 
separately during the preassessment phase and prepared a report. During the assessment phase of Part One, the 
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teams worked collaboratively to develop and administer the survey instruments for data from students and faculty. 
However, each team wrote separate final reports as part of the course requirements. A final exam and participation 
were part of each individual’s final grade. 

Part Two: Follow-on assessment. After the class was over, students could elect to continue with the 
project. In the fall 2009, the three doctoral students began working with the same instructor on gathering additional 
data; they are the co-authors of this paper. 



Data Sources and Instruments 

Data collected through various means and from the level one and two stakeholders was kept confidential 
and placed in a secure location. Any potential identifiers of individuals were removed and findings were expressed 
in aggregate form. 

Part One: Preassessment and assessment phase data sources & instruments 

Extant data. Extant data was provided by the course instructor. The documents showed course availability 
documents were from fall 2006 to summer 2009 included information on: how often courses were offered, whether 
they were online or on campus, number of seats available and number filled per semester/term. The enrollment data 
showed total number of students enrolled in the IDD program and by type of degree from spring 2004 to summer 
2009. Documents on degree requirements for Master’s, Doctoral Foundation Track and Doctoral were also 
examined. Lastly, the team had the published results from previous needs assessments conducted in 2002 and 2004 
and the PowerPoint™ presentation on the third needs assessment in 2007 (as previously mentioned in this paper). 

In the preassessment phase, one of the teams looked at other graduate programs in Instructional Design and 
Instructional Technology. Specifically, they examined Utah State, Texas A&M University, Capella University and 
the University of Southern Mississippi. 

Student questionnaire. The Student Questionnaire (SQ) was constructed by the two teams. It contained 
31 items within the sections of demographics such as their program, employment, etc., type of course delivery taken 
and preferred, questions about course availability, scheduling, and satisfaction in their program of study and 
progress toward completion of their degree. It was administered through Survey Monkey®. 

Faculty interview questions. The two teams also developed specific questions to ask faculty on an 
individual basis. There were 8 open-ended questions to about their perspective on the history of the ID program, the 
program in its current state, and whether they thought student needs were being met. The faculty would be able to 
inform us of the process that takes places to make adjustments to the class schedule as well as their opinions and 
recommendations for improving the IDD program. 

Part Two: Follow-On assessment instruments. 

In the fall of 2009 and early spring of 2010, the team conducted a follow on survey and two focus groups. 

Follow-On student questionnaire. The Follow-On Student Questionnaire (FOSQ) was a modified version 
of the SQ from Part One. The changes and additions included removing some of the demographic questions and 
asking questions about specific courses the students had or had not taken. The FOSQ was reviewed by the instructor 
prior to uploading to SurveyMonkey.com. The FOSQ was to be delivered to students enrolled in ID courses in fall 
2009, so that newly admitted and returning students could respond to it. We also allowed those who responded to the 
SQ from Part One to respond, but only to the new questions found in FOSQ. 

Focus group discussion agenda. The focus groups were conducted to further examine findings about 
online and on campus course offerings. The moderators used a list of seven questions. The discussion topics topics 
centered around what the ideal environment would be for students (online, on campus or both), their opinions on 
advising, the kinds of courses offered and if the focus group participants felt they were able to take the courses they 
needed. The focus groups were delivered via Second Life, on Jaguarland. Using Second Life for meeting with 
respondents, the conversation log of the focus group became the data source for later analyses. 
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Data Collection Procedures and Results 



For clarity, the Data Collection Procedures and Results for Part One of the NA will be discussed first, 
followed by the discussion Data Collection Procedures and Results for Part Two of the NA. 

Part One: Preassessment and assessment phase data procedures 

The instructor provided the documents on course availability, enrollments, and degree requirements at the 
beginning of the second week of the course to both teams. Additionally, she provided hard copies of two journal 
articles and uploaded the PowerPoint presentation to the course companion website for both teams to review and 
use. The two teams also gathered promotional brochures on the ID program and information on ID programs at other 
universities for comparison purposes. 

The SQ for Part One was developed together by both teams and reviewed by the instructor. Once changes 
were made, the questionnaire was delivered electronically using SurveyMonkey.com. One member of the team sent 
an email announcement explaining the purpose and requesting participation through the ID program’s listserv for 
faculty, students, and alumni. Location of the SQ and a time frame for responding was also included. 

To interview faculty, the teams worked together to develop the interview questions. The instructor again 
reviewed the items and the teams Members selected a faculty member to contact for an interview and made an 
appointment with them. Each interview was recorded and then transcribed. 

Results of Part One: Preassessment and assessment phase 

Results of extant data analyses. After examining the degree programs in Instructional Design and 
Instructional Technology at the University of Utah, Texas A & M, the University of Southern Mississippi and 
Capella University, the Master’s and PhD program at USA are comparable to these other universities as far as hours 
required for coursework, and in offering a mix of online and on campus courses. The exception is Capella 
University, which is an accredited online institution only. 

According to Clinton and Stevenson (2007), student enrollments were declining as early as 2003 and the 
trend continued as verified by the student enrollment records. Whereas the total number of students in fall 2005 was 
1 19, the total number in summer 2009 was 74. 

Other records also revealed that student enrollments had declined. In spring semester, 2007, 25 Master’s 
students were taking courses and only 17 were enrolled in summer 2009. The enrollment for doctoral track students 
in summer 2009 was 12. Percentagewise, the enrollment in the master’s program decreased by 23.5% from spring 
2004 to spring 2009. For doctoral students, enrollment fell 41.9% during the same time period. However, recall that 
the doctoral track program is a relatively new degree offered by the ID program; obviously, no data is available prior 
to fall 2005. Additionally, if data for doctoral track enrollments were combined with master’s student for summer 
2009, then it may more accurately reflect the decline with master’s students. Furthermore, doctoral track and 
master’s students are not required to be continuously enrolled; therefore, enrollment numbers may fluctuate 
somewhat. 

Additionally, analyses of other documents (i.e., course schedule documents) indicated two major areas of 
potential concern. First, it appeared that courses were offered in such a way that made it difficult for students to 
complete their degree in a timely manner. This included the scheduling of both on campus and online courses. 
Second, there seemed to be potential that if students were unable to take courses when needed in order to complete 
degree requirements, there may be dissatisfaction among the current student populations. This scheduling situation 
included core and support courses as well as research and statistics courses (Davidson-Shivers et al, 2004). 

Based on the previous needs assessment 2004 (Ingram et al., 2005) indicated that students in the master’s 
program responding to the questionnaire appeared to rate their satisfaction in course availability, content, 
scheduling, and the ID program lower than students in or graduates of the doctoral program. Because of the focus of 
the NA focus was on fact finding about the program students and alumni, the instrument used did not require 
respondents to explain why they held such views. 

Student questionnaire (SQ) results. The descriptive statistics from the SQ in Part One were computed 
using the built-in data analysis tools provided by SurveyMonkey.com. 

Based on the preliminary findings from the extant data, both NA teams perceived that students being 
unable to complete their degree programs in a timely manner. However, results based on the SQ data indicated that 
this might not be the case. 
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Twenty-five participants completed the questionnaire, with eight who identified themselves as graduates, or 
alumni, of the ID program. The remaining 18 were currently enrolled, which is an overall 24% return rate for current 
students. As shown in Table #, the vast majority respondents (96%) indicated that they were making satisfactory 
progress in their degree and they were satisfied with the student-teacher ratio within courses at the time and 85% 
were satisfied with faculty advising. With regard to course availability, only 14% indicated dissatisfaction. The 
majority (81%) also indicated their courses were a mixture of online and on campus course delivery; they indicated 
that they did this in order to graduate. Sixteen percent of those surveyed that they indicated a preference for online 
courses. The main reason was that their jobs prevented them from attending on campus classes. As for advising and 
scheduling,-the majority (84%) thought a two-year course plan would be helpful for them to plan their semesters in 
advance. 



Results of faculty interviews. The team members interviewed four of the ID faculty. Two faculty 
members stated that the declining enrollments in the Master’s program were a cause for concern. One professor 
stated that the program could no longer offer the number of support courses that were offered in the past due to the 
low numbers. Two professors responded that the ID program traditionally offered core courses at least twice a year. 
Additionally, they stated that with the[ relatively new] online master’s degree, the program was required to offer 
enough courses online in order for the online students to meet minimum number of credit hours requirements (i.e. 40 
graduate credit hours) for the master’s degree. Two professors thought there should be more courses offered online, 
however, converting an on campus course to an online course is time consuming and they needed more time to do 
that. 

All commented that offering a two-year calendar could help students plan and complete their Plans of 
Study and enable the students to graduate in a timely manner. The faculty also commented that students needed to 
take responsibility for these Plans of study and talking with their faculty advisors on a regular basis. One faculty 
member indicated that if there were enough students requesting a particular course, they could possibly offer it. 

Three professors responded the master’s program needs more required courses and fewer support courses. All four 
of the professors shared comments that they were satisfied with the requirements for the doctoral and doctoral track 
degree programs. 

In regards to reasons why the declining enrollment within the ID program were occurring, faculty 
responded with a variety of opinions. Two expressed a similar concern that a lack of recruitment was hurting the ID 
program overall. Other opinions by various faculty members were that the nationwide decline in university 
enrollments due to the economy, that lower numbers in the doctoral program may be due to it not being offered as a 
fully online program. One remarked that requirements changed for teachers pursuing the master’s degree which, in 
turn, made it less viable for them to enter in the ID Master’s. However, one faculty member indicated that 10 new 
students were expected to enroll in fall 2009 and was cautiously optimistic. This same person suggested that during 
times of economic downfall, many times students will get a graduate degree to increase their qualifications. 

In sum, although the findings from Part One suggested that students were satisfied with their program and 
they were able to make progress. Students who said they had to substitute courses for other intended courses in order 
to attain their degree was a new finding and a cause for concern. For instance, some students who planned to do their 
program fully online found it necessary to take classes oncampus. It was thought that while the findings were 
interesting, more/further clarification about online and on campus delivery was needed. It was thought that these 
other issues could be explored further. 

Part Two: Follow-on assessment data procedures 

The Follow-on team met to discuss next steps at the beginning of fall semester. This Follow-On 
Assessment experience was a “learning opportunity” for two of the students; one student was assigned a primary 
lead in facilitating the study as part of a directed study course. 

The three team members added questions based on the Part One findings and the suggested areas that could 
be further explored. For the FOSQ, they removed some demographic items that were found to be either unnecessary 
or allow for the identity of the respondent to be known. Once approved, potential student participants were notified 
about the location of the FOSQ with an explanation of the purpose using the ID listserv. The FOSQ questionnaire 
was delivered using SurveyMonkey.com in November 2009. 

After the FOSQ was completed, potential participants were again notified of the Focus Group’s time and 
location and its purpose through the ID listserv. As stated previously, the focus group was conducted in Second Life. 
Although the moderator, one of the follow-on team, and respondents were within speaking distance, the virtual 
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environment made it easy for the moderator to log the conversation for later review and analysis of information 
collected. 

Due to the timing of the focus group in the fall 2009 only a few participants attended, the Follow-On team 
conducted a second focus group at the start of spring 2010 term using Second Life. However, this time, the 
participants were in different physical locations. The questions used in the focus group were similar to the 
questionnaires but the participants could discuss the issues more in depth. 

Results of Part Two: Follow-On Assessment 

Results of FOSQ. As with the SQ in Part One, descriptive statistics from the FOSQ were computed using 
the built-in data analysis tools provided by SurveyMonkey.com. 

Overall, the results from the FOSQ supported the findings of the Part One needs assessment. Of the 22 
students who completed the questionnaire, 20 felt that they were making satisfactory progress in the program. One 
respondent selected “lack of choice of courses” and “personal responsibilities” as the reasons for lack of were 
satisfactory progress toward completing his/her degree and one student skipped the question. 

Seventy two percent of respondents indicated that when meeting with their advisors, they discussed their 
plans of study more than anything else. Of the 72%, approximately 33% indicated that they met with their advisors 
once per semester, but another 38% indicated that they met with their advisors two or more times per semester. 

As anticipated, the results indicated that 81% of the respondents felt that a set course schedule would be 
beneficial when considering their plans of study. This finding supported the identified need that a scheduling 
problem existed in which students were unable to plan their courses far enough in advance. 

In the FOSQ, a list of courses was provided; respondents were asked to indicate two things: a) the courses 
that they were interested in taking and b) if they had taken it. Respondents marked both categories for a majority of 
the courses. However, we noticed a difference in responses respective to four courses about technology tools. 
Between five and nine students identified courses as interesting to take, but only three have take the animation 
course and only one had taken the course on emerging technology. 

Based on the findings showing discrepancies for four technology tools courses, we could have assumed the 
courses were offered at an inconvenient time/day during the semester or not in the semester that students needed it. 
However, we referred back to extant data from Part One to discover that the four courses had not been taught in a 
few years. Therefore, this finding supports the concern that certain courses are not offered frequently enough to meet 
student interest and needs. 

It was anticipated by the Follow On team that if students were dissatisfied with course availability, it would 
be with the availability of online courses instead of on campus courses. Of the 16 respondents that addressed course 
availability, 1 1 indicated that they were satisfied with the availability of online courses, and seven indicated 
satisfaction with the availability of on campus courses. Therefore, this finding does not appear to support that there 
is a need to address any potential imbalance in the numbers of online courses and on campus courses. 

Results of the focus groups I and II. Participants in the focus group meetings had taken the questionnaire. 
The purpose of the focus groups was to follow up on questions related to the FOSQ data. In Focus Group I, five out 
of six participants indicated that their preference was to take both on campus and online classes rather than one or 
the other. The participants commented that online classes allowed more flexibility in schedule and were appropriate 
for project-driven classes. The group also unanimously agreed that many professors were not trained to teach online 
and that there was a trend toward professors of online classes to withdraw from interaction with students and not 
“build community.” Furthermore, their discussion suggested that in online classes that were not project-based, the 
lack of interaction could be problematic for them. 

When asked what courses they would like to see as part of the program, two focus-group participants 
mentioned that they would like to see courses address gaming, animation, and simulations, which one participant 
thought the program “no longer offer[ed].” This comment again reflects a confirmation of FOSQ finding that 
respondents expressed an interest in taking a class, but for some reason did not actually take it. Three Focus Group I 
participants also agreed that they would like to see additional qualitative research courses. 

In Focus Group II ( n = 2), the participants agreed that they liked taking both on campus and online classes. 
According to one participant, “on campus classes allowed for free exchange of thought among students and the 
professor [and] are more preferable for theory-driven classes.” Another participant indicated a preference solely for 
on campus classes and stated that an on campus class was “worth the drive.” 

One participant in the second focus group met with their advisors at least twice a semester about various 
topics, including course planning. The other participant who did not meet as frequently was a new student and had 
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not chosen a permanent advisor. Both focus groups agreed that that a two-year course schedule would be helpful in 
completing their plans of study. This seemed to indicate that it would allow them to make plans far in advance rather 
than waiting to see what was offered during each semester. 

Finally, the focus group participants were asked how the program could improve their recruitment efforts. 
The participants indicated that they were not aware that the program made recruitment efforts. Only one person 
mentioned seeing a poster in the Communications Department. Other than that, the rest of the participants heard 
about the ID program through their friends and former advisors. 

Conclusions 

At the start of the needs assessment project, there were indications that graduate students in the ID program 
were not able to complete coursework in a timely manner. It was suspected that classes were not available when the 
students needed them; and some students had difficulty taking courses that were only offered on campus. 

After collecting data from various sources in both Part One and Two, we determined that the ID program 
was meeting students’ needs overall, but there were some areas of concern. First, course planning and scheduling 
was an issue in that students were unable to plan their courses far enough in advance. Faculty members had a 
general idea of what would be taught from semester to semester and discussed this when meeting students, but 
students were not always aware of this fact, especially if they did not meet with their advisors on a regular basis 
each semester. 

Second, certain support courses were not offered frequently enough. Offering support courses, including 
technology tools for the master’s degree, could allow students additional knowledge and skills that they viewed as 
interesting. 

Third, we observed from the findings that the ID program needed to increase its recruitment efforts. At the 
very least, recruitment would increase awareness of the ID program and its field. At best, there could be an influx of 
new students, which, in turn, could boost enrollment in courses and, thus, increase student demand for other courses 
to be offered on a regular basis. 

As a final note: The ID program approved and implemented a curriculum for master’s level and doctoral 
courses for 2010-2011. The courses are sequenced over a two-year period to assist students in planning and 
scheduling courses. The curriculum was provided to students through the ID listserv in March 2010 and again every 
term. 
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Abstract 

The increased accessibility of technology and internet connections has enabled organizations to provide 
their workforce the opportunity to engage in distributed education. Harnessing this innovation calls for 
organizational and technological infrastructures that support the interplay of knowledge and knowing” (Cook & 
Brown, 1999, p. 381). This paper explores the evidence of knowledge convergence in online knowledge based 
CoPs. Learning outcomes assessed declarative knowledge, convergence, and knowledge application. A comparison 
group (self-paced design) was used to draw conclusions about the differential effects of knowledge-building 
strategies on these learning outcomes. The results show there was a difference in pretest/posttest scores, and 
positive evidence of knowledge convergence. Our findings pointed to higher posttest scores and higher level of 
convergence in the online CoP strategy. 



Introduction 

Today, knowledge is viewed as a commodity, which enables organizations to successfully compete on a 
global scale. Organizations value an individual’s knowledge as evidenced by corporate use of knowledge 
management systems to efficiently capture and catalog explicit knowledge. Cook and Brown (1999) argue that the 
focus on organizational knowledge, and management of knowledge, is based on a traditional understanding of the 
nature of knowledge. The increasing proliferation of technology and the Internet have enabled individual and group 
learning opportunities to transform tacit knowledge into explicit knowledge through online communities of practice. 

Tacit knowledge is the result of knowing and utilizes explicit knowledge through practice. Tacit 
knowledge is present in work and workplace learning and is described as ‘know how’ transferred by storytelling, 
conversation, and narrative. Orr’s (1999) research illustrates the importance of tacit knowledge to organizations 
where Xerox’s formal training programs and instruction manuals failed to provide relevant information for the 
service technicians to perform their jobs. In such instances, acquiring a tool (knowledge) is not the same as being 
able to use it (Brown and Duguid, 2001). 

Explicit knowledge is easily articulated and takes the form of documents, websites, customer relationships 
management (CRM) databases, and manuals that can be shared, and transferred to others. Corporations construct 
explicit forms of knowledge such as formal procedures, manuals, training and job descriptions that in most instances 
do not relate to authentic employee practices. Cook and Brown, (1999) suggest that explicit knowledge is regarded 
as formalized and can be documented. They further argue that focusing on an individual’s explicit knowledge is 
insufficient; rather, it is important to focus on individual and group tacit and explicit knowledge contexts in 
organization learning. 

In the workplace, explicit knowledge is used as a management tool to influence and control organizational 
knowledge. While manuals are helpful to document knowledge for the organization, explicit knowledge is 
dependent upon tacit knowledge to be truly effective. There is a need for organizations to go beyond the 
documentation of formal, explicit knowledge to reveal and reify the ‘knowing’ dimension of knowledge. The 
challenge for organizations is to focus on the relationship between explicit and tacit knowledge building so that an 
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individual’s or group’s experiences are reified or hardened into a more explicit form. “Harnessing this innovation 
calls for organizational and technological infrastructures that support the interplay of knowledge and knowing” 
(Cook & Brown, p. 381). 

A knowledge-based community of practice is a type of learning environment intended to codify and convert 
valuable, tacit knowledge into explicit knowledge. The reification process results in a collection of permeable best 
practices that can be shared by others in the community of practice. Knowledge based CoPs are the vehicle in which 
its passengers are able to propel the advancement of collective knowledge to develop individual skills and practices 
by achieving full participation of the members. The following section details the characteristics of knowledge-based 
communities of practice. 

Research on the design and development of online communities of practice is still emerging. Among the 
issues is that “the evolutionary pattern of CoP development is poorly understood” (Schwen & Hara, 2003, p. 262). 
Schwen and Hara suggest that communities of practice are fully functioning when they evolve over time, which 
makes them difficult to study. Questions also have been raised about the role of knowledge in communities of 
practice. Researchers have varied interpretations of knowledge, particularly with regard to knowing in practice, and 
hence is worth studying. Schwen and Hara (p. 263) state “knowledge and knowing epistemologies are two distinct 
processes that require different designs to support optimal community learning.” 

The purpose of this study was to investigate the effectiveness of varied knowledge building principles, i.e., 
idea improvement, real ideas, authentic sources, and community dialog when used in an online community of 
practice learning environment to promote the convergence, reflection, and sharing of knowledge that can be reified 
and incorporated into a learning organization’s practices. This study investigated the following research questions: 



1. What are the effects of learning environment design on pre and posttest scores of declarative 
knowledge? 

2. What are the effects of learning environment design on employee’s knowledge convergence as 
measured by criterion essay assessment? 

3. What is the impact of learning environment design on individual on-the-job performance, as measured 
by customer survey performance indicators? 

Method and Procedures 



Participants and Context 

The participants for this study included a sample selected from 1001 dealership Service Advisors from 
Subaru, of America, Inc. who are required to complete the Building Service Excellence online training course to 
attain customer service certification and qualify for the Service Advisor SUMMIT Recognition incentive program. 
Within the dealership organization, the Service Advisor has the most customer contact; therefore, he or she is pivotal 
in creating and maintaining customer loyalty. The participants were comprised of 68% male and 32% female. All 
participants graduated high school with 24% reporting some college courses. The experience level of study 
participants ranged from less than six months to over 35 years. 

The seven-hour classroom-based training course entitled Building Service Excellence was converted into 
two different treatments for this study: (a) an online community of practice learning environment and (b) an online 
self-paced instruction environment. Building Service Excellence is the cornerstone customer service course and 
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required of all Subaru dealership personnel. The course material introduced Service Advisors to basic customer 
service skills that are essential to maintain and increase customer loyalty. 

To be consistent across treatments, all instructional content material and assessments are identical. The 
online community of practice learning environment and the self paced instructional environment Building Service 
Excellence courses contain seven customer service related topics: 1) Personal and Practical Needs 2) Steps to 
Service 3) Key Principles 4) Taking the HEAT 5) Walkers and Talkers and 6) Recovery and 7) Action Planning. 
The presentation of the instructional materials is identical utilizing WebEx Presentation Studio to produce 
PowerPoint-like presentation instructional modules with audio narration, Adobe Captivate modules, text and 
electronic documents. The two treatments of the Building Service Excellence course were developed and designed 
to operate in Moodle (Modular Object-Oriented Dynamic Learning Environment). Moodle is a free open source 
Learning Management System application developed and maintained by a consortium of educators to promote 
constructivist pedagogy. Moodle’s functionality offers many features for the design of online instruction. 
Instructional designers have the ability to create comprehensive, content-rich, highly collaborative learning 
environments. The following table shows Moodle features. 

The study was conducted online using the Learning Management System, Moodle. The study was 
available to approximately 1200 Subaru dealership Service Advisors across the United States. The treatment 
materials were available 24/7 from any computer; from either work or home environment. Computer hardware 
varies from student to student; however, each PC is equipped with a CD drive, mouse, sound card, flash software, 
and a high-speed internet connection. After students enrolled in the course, they took the same pretest. They then 
proceeded through the course material. The participants enrolled in the knowledge-based course proceeded through 
the course material at their own rate. The participants in the community-based course followed a course timeline 
that lasted 12 weeks. The material was delivered at the student’s convenience and was a requirement for 
qualification in Subaru’ s Service Advisor SUMMIT incentive program. The study was conducted over a six-month 
period. There were no restrictions on the amount of access attempts, time limitations, or location once students first 
access the course. After the course ended, participants were asked to take a knowledge posttest and an essay 
posttest. After 60 days, SOLI scores were collected for participants by an automatic data processing script and 
exported into an Excel spreadsheet. 



Results and Discussion 

The results of the study are presented in the following sections according to each research question. 

Declarative Knowledge Pre-Test 

The pretest data were analyzed to determine if the two sample means were similar. Table 1 shows the 
descriptive statistics for pretest scores from the self-paced and knowledge based community-learning environments. 

Table 1: Descriptive Statistics for Self-Paced Pre-Test Scores vs. Community Pre-Test 





N 


Mean 


SD 


SE Mean 


Self-paced Pre-Test 


61 


23.78 


2.75 


0.35 


Community Pre-Test 


61 


23.42 


2.82 


0.36 


Difference 


61 


0.364 







The pretest scores of participants from each treatment were analyzed to determine comparable population 
means. To test the null hypothesis that there is no difference between the pretest scores of the self paced and 
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community learning environments, a two-tailed test using the t-test statistic was used to analyze variances and 
means of the two populations. Glass and Hopkins (1984) state that there are three t-test assumptions: “(1) the X’s 
within each of the two populations are normal X’s y distributed; (2) the two population variances are equal; and (3) 
the individual observations (X’s) are independent (Glass & Hopkins, 1984, p. 236). It is unknown whether the 
observed value will be greater or less than expected and a two-tailed t-test is a more conservative or demanding 
(Creswell 2005, p. 188) analysis on the data. The level of significance or p value is (p<.05). Table 4 shows the two- 
tailed t-test statistic for the self-paced and community pretest. 

Table 2: Two-Sample T-Test for Self-Paced Pre-Test Scores vs. Community Pre-Test 



Variances 


t- 

value 


df 


SE of 
Difference 


95% Confidence Interval 


/i-value 










Lower 


Upper: 




Equal 


.72 


120 


0.364 


-0.634 


1.361 


0.472 



The t-test revealed equal means on the self-paced and community pretests as indicated in table 2 ( t (120) = 
.12,p<A12). Thus, there was no difference between pretest scores across groups and accept the null hypothesis. 

It was expected that the Service Advisors would score higher on the posttest than the pretest across 
treatments. Additionally, it was expected that the Service Advisors who self-selected the knowledge building 
community intervention would have higher posttest scores than the self-paced instruction group. To address this 
research question, analyses of the sample’s posttest scores were investigated. The primary data sources for 
addressing this question came from the analyses of posttest scores from the self-paced treatment and the posttest 
scores from the knowledge building community. 

Declarative Knowledge Posttest 

The multiple-choice posttests were administered after the completion of the content for the self-paced and 
knowledge-based community treatments. The researcher expected that Service Advisors in the knowledge -based 
community learning environment treatment would score higher. 

The sample is considered large; greater than 30, therefore an assumption can be made that the scores are 
normally distributed. Table 3 shows the descriptive statistics for the posttest scores in the CoP and self-paced 
learning environments. 

Table 3: Descriptive Statistics for Self-Paced Post-Test Scores vs. Community Post-Test 





N 


Mean 


SD 


SE Mean 


Self-paced Post-Test 


61 


25.88 


1.78 


0.23 


Community Post-Test 


61 


28.638 


0.664 


0.085 


Difference 


61 


-2.757 







To evaluate the statistical significance of the difference in test scores, a two-sample t-test was performed. 
Table 4 reports the results of these tests and shows that participants in the community of practice learning 
environment had higher scores by a statistically significant margin ( t (120) = - \ \3\,p< 0.00). 
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Table 4: Two-Sample T-Test for Self-Paced Post-Test Scores vs. Community Post-Test 



Variances 


t-value 


df 


SE of 
Difference 


95% Confidence Interval 


/>-value 










Lower 


Upper: 




Equal 


-11.31 


120 


-2.757 


-3.240 


-2.274 


0.00 



Knowledge Convergence 



Essay Assessment 

The case-based essay assessment was the final measurement completed by the learner. Each of the five 
scenario based essay questions required the learner to read an authentic workplace situation and respond in writing 
using storytelling to integrate course content and practical workplace experiences. The techniques used in this study 
attempt to measure unobservable theoretical constructs using concept mapping to determine knowledge 
convergence. 

Clariana and Koul (2004) suggest that there is a natural relationship between concept maps and essays, and 
content and process knowledge. Essays can be converted into concept maps to determine sharedness among the two 
groups and compare “group” mental models. To analyze the essay scores, the researcher utilized a method called 
Analysis of Lexical Aggregates (ALA-Reader, a software application) that enables the conversion of written essay 
responses into concept maps. This text representation tool analyzes small text samples using specific key terms. 

The key terms selected and shown in Figure 1 were derived from the course content. 



customer(s) 


customer 








car 


car 


vehicle 






loaner 


loan 


rental 






need(s,ed) 


need 








repair(s) 


repair 








shuttle 


shuttle 


transportation 


ride 


pickup 


complete 


complete 


done 


ready 




time 


time 








appointment 


appointment 








available 


available 








offer 


offer 








wait(ing) 


wait 








inconvenience 


inconv 








contact 


contact 


phone 


call 




take 


take 








service 


service 








apologize 


apolo 


sorry 






make 


make 








know 


know 








what 


what 








let 


let 








today 


day 








explain 


explain 








thank 


thank 








may 


may 








give 


give 








ask 


ask 








situation 


situation 








possible 


possible 








first 


first 









Figure 1: Key terms derived from essays 
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Key terms include synonyms and metonyms at the sentence level and saves the data into a link array file. 
Clariana, Wallace and Godshalk (2009) use further analysis of sentences in text by eliminating all key terms except 
for the pre-selected key terms derived from the expert essay. The key terms that co-occur in the same sentence of 
the written responses are included in the proximity array. The link array as shown in Figure 2 can be further 
analyzed by Pathfinder network scaling. The proximity arrays can be displayed as force-directed graphs. These 
graphs are node-to-node representations that depict key word propositions in the text as a graph (Clariana et ah, 
2009). 

0 
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00000000000000000000000000000 

Figure 2: Example of Link Array 

After the essays were collected, they were manually spell checked using MS Word, then word frequency of 
the entire essay corpus was analyzed using http://textalyser.net (see Figure 3). The frequency analysis output was 
used to create a terms.txt file containing a list of the 30 key terms and their synonyms and metonyms. Then ALA- 
Reader software utilized the terms.txt file to process (using text pattern matching) all 121 participant essays, 
generating a proximity file (prx) for each essay. 




Next, the 121 prx files were randomized within treatment and numbered. To form two groups for each 
treatment for comparison purposes, Pathfinder KNOT software was used to average the odd numbered (e.g., 
1..3..5..x) and the even numbered (e.g., 2. .4. .6. . .etc.) prx files, and then KNOT was used to convert each prx file into 
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a pathfinder network (PFNET). This approach produced an expert essay PFNET (see Figure 4 — 6) and four ‘group’ 
PFNETs including SP 0 dd, SP even , CoP od d, and, CoP even . 




Then KNOT software was used to compare these five PFNETs. Examination of links in common and 
percent overlap indicate that the CoP PFNETs were more alike (66% overlap between the even and odd CoP groups) 
relative to the SP PFNETs (33% overlap between the even and odd SP groups). 

Research suggests that using ALA-Reader and KNOT for analyzing an individual essay with an expert 
essay count the common network links and that the common measure is a better predictor of knowledge 
convergence because incorrect associations are disregarded (Taricani & Clariana, 2006). Since these ‘group’ 
PFNETs are derived from the individual students’ essays, among other things, we conclude that the online 
community of practice strategy (CoP) led to a convergence of these participants mental models that was reflected in 
their essays and thus in the group PFNETs. 

Work-related Practice 

The SOLI score data were analyzed to determine if the two sample means were different. Table 5 shows 
the descriptive statistics for customer service survey SOLI scores from the self-paced and knowledge based 
community learning environments. 



36 



Table 5: Descriptive Statistics for Self-Paced SOLI Scores vs. Community SOLI 





N 


Mean 


SD 


SE Mean 


Self-paced SOLI 


61 


93.11 


3.77 


0.48 


Community SOLI 


61 


97.34 


2.99 


0.38 



A two-tailed t-test was calculated for determining whether the difference between two independent sample 
means represented a true difference between populations thereby rejecting the null hypothesis of no difference 
between the means of the population (Shavelson, 1996, p. 339). Table 6 represents the two-sample t-test and the 
confidence interval for the self-paced SOLI scores and the knowledge based community SOLI scores. The t-test 
revealed unequal variances on the self-paced and community posttests as indicated in Table 6, with the community 
group outperforming the self-paced group. Table 6 reports the results of these tests and shows that participants in 
the community of practice learning environment had higher scores by a statistically significant margin (t(l 13) = - 
6-86, p < 0.00). While these data were not conclusive as to direct impact, they nonetheless support the possibility 
that participation in the knowledge based CoP learning environment was associated with individual performance 
gains of those who participated. 

Table 6: Two-Sample T-Test for Self-Paced SOLI Scores vs. Community SOLI 



Variances 


t-value 


df 


SE of 
Difference 


95% Confidence Interval 


/7-value 










Lower 


Upper 




Equal 


-6.86 


113 


-4.227 


-5.447 


-3.007 


0.00 



Closing 

In summary, participants were able achieve knowledge convergence as the result of collaboration in a 
community of practice learning environment. Social learning takes place when people share a like interest, 
communicate, and collaborate over time to build knowledge. More recently, communities of practice have been 
linked with knowledge management as organizations recognize their potential contributions to human capital and 
organizational performance. Knowledge based CoP learning environments have the ability to drive corporate and 
initiative strategy, create new products and services, communicate exemplary practices, which decrease the learning 
curve of novices, solve problems quickly, and develop practical skills. Within knowledge based CoPs there is a 
tight connection between knowledge and activity. One implication for the design of knowledge-based communities 
of practice is how to develop and support highly collaborative activities to achieve knowledge convergence. 
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Abstract 

This research was designed to explore the impact of a wiki-based international collaboration project on 
students’ 1) cultural competencies, 2) comfort using technology, including Web 2.0 tools, to interact and collaborate 
with unknown colleagues, and 3) perceptions of ideas for using technology, including Web 2.0 tools, in their future 
classrooms. Survey results showed significant changes in cultural competency and perceived ideas for using 
technology from pre- to post-semester while interview data provided support for, and greater insights into, survey 
results. Furthermore, students’ perceived comfort for participating in Web 2.0-based international collaborations was 
the best predictor of cultural competency. Implications for the integration of international collaborations into on- 
campus courses, using Web 2.0 technologies, are discussed. 

Introduction 

We live in a world characterized by globalization, emerging new technologies, and a knowledge-based 
economy. Together, these characteristics impact every aspect of our society including communication channels, legal 
systems, socio-cultural trends, and educational approaches (Olson & Olson, 2000; Teasley & Wolinsky, 2001). 
According to West (2010), the success of our future graduates depends on their abilities to be both culturally and 
technologically competent as they work and interact with diverse, geographically dispersed people, using a variety 
of technology tools. 

Bielefeldt and High (2007) define cultural competency as “the ability to effectively interact with people 
from diverse cultures and recognize the importance of cultural differences” (p. S2-G5). As early as 1969, educators 
have suggested that students who are preparing to be teachers should engage in cross-cultural experiences in order to 
enrich and expand their understandings of “a culture different from the one in which they were bom and raised” 
(Taylor, 1969, p. ix). More recently, a number of professional organizations have stressed the importance of 
producing culturally competent teachers. For example, in their most recent set of standards, the International Society 
for Technology in Education (ISTE, 2008) emphasized the need for K-12 teachers and students to participate in 
global learning communities and to demonstrate “cultural understanding” and “global awareness” (Standard 4.d). 

In response to the call to provide increased cross-cultural experiences for undergraduates, universities have 
sought to expand their study abroad programs and to provide greater access to international opportunities. 
Unfortunately, very few students actually participate in these programs due to time or cost constraints (Bellamy, 
2006). If all of our pre-service teachers are to benefit from cross-cultural experiences, educators need to provide 
alternative approaches that decrease traditional costs, while maintaining, or even strengthening, traditional benefits. 

Web 2.0 applications have the potential to address these needs by engaging students with global partners 
using a variety of web-based tools. These technologies can play an increasingly important role in the development of 
“communities of practice” (i.e., where individuals with a common set of problems interact to devise solutions), 
especially when group members are globally distributed and face-to-face time is limited (Wenger, White, Smith, & 
Rowe, 2005). Wikis, particularly, offer a potentially effective platform for facilitating cross-institutional and cross- 
cultural interactions; as web documents that are automatically published, they are easy to both edit and share (see 
Table 1). According to Bonk, Lee, Kim, and Lin (2009), “wiki-related projects provide opportunities for learning 
transformation when they expose learners to new points of view or perspectives” (p. 126). Crucial to this 
transformation, however, are the interactions that occur among participants to facilitate development of the wiki 
product. 
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Table 1. Wiki properties enabling cross-cultural collaboration 



Wiki Property 


Resulting Effect 


Web 2.0 technology 


Familiar to, and within reach for, non-tech savvy 
students 


Document co-editing 


Easy to asynchronously produce content 


Automatic publishing 


Easy to share, exchange, and access material 


Non-hierarchical 


Student-centered and owned workspace 



Although educators (Bonk et al., 2009; Chia, Poe, & Wuensch, 2009; West, 2010) are advocating the use of 
wikis as an effective means for facilitating cross-cultural collaborations, little is known about the impact of these 
activities on students’ cultural competencies. To what extent can short-term participation impact students’ ideas 
about working with diverse others? Although wikis make interaction among culturally diverse and geographically 
dispersed students more feasible, in order to truly benefit, participants must actively collaborate to achieve a 
common goal. According to Larusson and Alterman (2009), the key to students’ success is the ability to effectively 
1) communicate a shared understanding of the purpose of the product (e.g., wiki chapter), and 2) coordinate the 
activities, roles, and responsibilities required to create it. Unfortunately, these functions are difficult to accomplish in 
online communities, particularly among students who are new to both wilds and cross-cultural collaborations. 

To explore these ideas, a wiki-based collaboration project was developed as a five-week unit within an 
introductory educational technology course in order to provide an international experience for both pre-service 
teachers from the United States and their partners from England, Russia, South Korea, and Sweden. The primary 
goal was to involve local and international students in the co-design and co-creation of a wiki repository, focused on 
Web 2.0 technologies for teaching and learning. 

Purpose 

This research was designed to determine the efficacy of using a shared wiki to engage students in 
international collaborations. More specifically, we aimed to determine the impact of this approach on students’ 1) 
cultural competencies, 2) perceived comfort engaging in collaborations with distant others, using technology, and 3) 
perceived knowledge for using wikis (and other technologies) effectively in their future classrooms. Additionally, we 
were interested in examining the relationships among these variables to determine the extent to which changes in 
cultural competency could be predicted from other defined (perceived comfort; perceived knowledge) and/or 
demographic (age, gender, class) variables. 



Methods 

We used a mixed methods research design to examine the impact of this approach on undergraduate 
students enrolled in an introductory educational technology course at a large Midwestern university. To help us 
answer our research questions, survey and interview data were collected during Fall 2009. Data from project 
narratives were used to triangulate survey and interview findings. 

Description of Site and Participants 

Participants included 202 pre-service teachers enrolled in a required 2-credit educational technology course 
in Fall 2009. The majority of the students were female (66.5%), first (37.9%) or second-year (37.4%) students, 
studying to be elementary (62%) or secondary teachers (24%). On a scale from 1 to 5 (with 1 being extremely low 
and 5 being extremely high), students’ pre-course ratings of perceived computer skills averaged 3.34 (i.e., “I feel 
comfortable completing most basic tasks on the computer.”). 

Students met weekly for a one-hour whole-class lecture and a two-hour lab, with 1 8-24 students assigned to 
each of 1 6 lab sections. Lab teaching assistants (TAs) divided students into smaller teams composed of seven or 
eight members (approximately three teams/lab for a total of 41 project teams). In addition, each team was paired 
with two to four international students from England (9), Russia (24), South Korea (64), or Sweden (20). Each team 
was tasked with creating a wiki chapter about a specific Web 2.0 tool (Twitter, Webnode, Mindomo, etc.). [A 
complete list of the Web 2.0 tools that have been investigated, to date, can be found at: http://web2insite.com/] 
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Procedures 

The project occurred during the second half of a semester long course and lasted approximately five weeks. 
Prior to the start of the project, lab TAs selected one member from each team to serve as the project manager (PM). 
The PM was responsible for coordinating team efforts, attending weekly project meetings, and disseminating 
relevant information to other team members. At the beginning of the project, team members divided up the various 
roles (e.g., researchers, designers, writers, lesson plan developers, wiki developers, evaluators) needed to complete 
the project. Although all members were encouraged to assume a primary role for one task, they often played several 
additional roles to ensure all tasks were completed successfully. International partners were encouraged to volunteer 
for roles they preferred and/or to contribute to any required tasks. 

Each team investigated a specific Web 2.0 application and created a wiki chapter (using Confluence wiki 
software) describing the application and its potential educational uses. As a cumulating experience, teams presented 
posters of their Web 2.0 applications at a Showcase Event, which was attended by their peers, local teachers, college 
faculty, and the general public. 

Data Sources 

Of the 3 16 students enrolled, 202 completed all pre-post survey measures: 1) a pre and post-survey 
measuring perceptions of cultural competency (Miville-Guzman Universality Diversity Scale, Short Form [M- 
GUDS-s ]; Miville et al., 1999), 2) a pre and post-survey measuring students’ perceived levels of comfort for 
participating in an international collaboration using Web 2.0 tools (Perceived Comfort Survey ), and 3) a pre and post 
survey measuring students’ perceptions of their pedagogical knowledge for using technology tools for teaching and 
learning (Perceived Knowledge Survey). 

The original M-GUDS survey was developed by Miville and her colleagues (1999) to measure students’ 
appreciation for similarities and differences in others, as evidenced by behavioral, cognitive, and affective 
components. This initial scale included 45 items; based on the results of an exploratory factor analysis, the total 
score (as opposed to subscale scores) was considered the most representative score to report, due to the uni- 
dimensionality of the scale. Reported alpha coefficients from early pilot studies ranged between .89 and .95 (Miville 
et al., 1999). The original scale was subsequently modified (Fuertes, Miville, Mohr, Sedlacek, & Gretchen, 2000) 
into a shorter version, M-GUDS-s, which consisted of 15 items, divided into three subscales: Relativistic 
Appreciation (cognitive; e.g., “Knowing how a person differs from me greatly enhances our friendship.”), Comfort 
with Differences (affective; e.g., “It’s really hard for me to feel close to a person from another country.”), and 
Diversity of Contact (behavioral; e.g., “I often listen to music of other cultures.”). (Note: Items on the affective 
scale were reverse coded.) Students are asked to rate their levels of agreement with each item on a scale from 1 
(strongly disagree) to 6 (strongly agree). Fuertes et al. reported inter-item correlations for each subscale ranging 
from .59 (Relativistic Appreciation) to .92 (Comfort with Differences), while the total score had an inter-item 
correlation score of .77. For our study, alpha coefficients were calculated; subscale coefficients ranged from .76 
(Comfort with Differences) to .82 (Diversity of Contact). The Relativistic Appreciation subscale had an alpha 
coefficient of .80; the alpha coefficient for the total score was .85, suggesting moderate - high reliability. 

The Perceived Comfort Survey consisted of six items for which students rated their perceived levels of 
comfort (from 1-strongly disagree to 6-strongly agree) for participating in an international collaboration using Web 
2.0 tools (e.g., “7 am comfortable interacting - through Facebook, etc. - with people Fve never met face-to-face.”). 
A calculated alpha coefficient of .80 suggested that the survey was moderately reliable. 

The Perceived Knowledge Survey consisted of seven items that assessed students’ perceptions of their 
pedagogical knowledge (from 1-strongly disagree to 5-strongly agree) for using technology tools for teaching and 
learning (e.g., “ I have specific ideas about how to use technology’ as an effective teaching tool. ”). The calculated 
alpha coefficient for this scale was .95, suggesting high reliability. 

Focus groups interviews were conducted with 25 project managers, as well as 11 selected project teams in 
order to understand the successes and difficulties encountered throughout the project. Sample questions included, 

“ What is the most important thing you will take from this project? For example, what was most surprising? Or what 
was most disappointing? What were some of the things you struggled with throughout the project as an individual or 
as a group? If you had a chance to do the project again, what would you do differently?” To analyze our data and 
answer our research questions, paired t-tests, correlations, and a multiple regression were used. Focus group 
interview data were analyzed using a simple pattern-seeking method to identify commonalities among students’ 
responses. 
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Results 



Changes in Cultural Competency 

A two-tailed paired t-test (df= 201) indicated significant increases on two M-GUDS-s subscales, Comfort 
with Differences ( t = -1.94; p = .05) and Diversity of Contact ( t = -6.77; p < .000), as well as on the total score (t= - 
3.64; p < .000). There were no significant changes on the Relativistic Appreciation subscale from pre to post- 
semester (see Table 2). Students’ judgments on the affective subscale, Comfort with Differences, increased from a 
pre -project mean of 4.37 (SD=. 69) to a post-project mean of 4.5 (£D=69); judgments on the behavioral subscale, 
Diversity of Contact, increased from a pre-project mean of 3.75 (£0=1.3) to a post-project mean of 4.48 (£D=,81). 
The total M-GUDS-s score changed from 4.08 (50= 57) to 4.31 (50=.64). 



Table 2. Pre- and Post-Survey Means and SDs on M-GUDS-s Assessment of Cultural Competency 



N=202 


Diversity’ of 

Contact 

(behavioral) 


Relativistic 

Appreciation 

(cognitive) 


Comfort with 

Differences 

(affective) 


Total Score 


Pre-survey Mean (SD) 


3.75 (1.30) 


3.85 (.76) 


4.37 (.69) 


4.09 (.57) 


Post-survey Mean (SD) 


4.48 (.81) 


3.94 (.92) 


4.50 (.69) 


4.31 (.64) 


Paired-t statistic 


-6.77 


-1.04 


-1.94 


-3.64 


p-value (2-tailed) 


.00 


.30 


.05 


.00 



These findings suggest that, through this project, students’ intentions to engage in, as well as their actual 
participation in, cross-cultural activities increased. Perhaps the positive experiences they had working with 
international partners on this project provided new appreciation for diverse cultures and helped them become more 
interested in, and more comfortable, participating in cross-cultural events and/or engaging in cross-cultural 
collaborations. In addition, students seemed particularly amenable to incorporating similar types of interactions in 
their future classrooms as noted by a number of comments in students’ project reflections. For example: 

1 think that the use of Web 2.0 tools and having my students collaborate with international partners 
would be beneficial to enhance the learning environment in the classroom. This gives the students 
the opportunity to not only get hands on experience with technology in class, but also to work with 
students in a different culture (Team member, Prezent.it). 

1 felt like this was a great learning experience and I think that my students could take a lot away 
from interacting with people from other countries. This can broaden a student’s thought process 
and cultural understanding (Team member, Mindomo). 

In general, students began the course with relatively high affective scores; the average score on the Comfort 
with Differences subscale was noticeably higher than the other two subscales (i.e., at least .5 point). Still scores on 
this subscale showed a relatively small but significant increase from pre- to post survey, suggesting that students’ 
initial positive attitudes toward working with diverse others were enhanced through their interactions with their 
international partners. Although students were quick to point out some of the difficulties involved in connecting 
with others across time zones, cultures, and geographical distances, they were also aware of the potential benefits. 

As described by a Cellblock team member in his project reflection: 

When working with the international partners I learned that there are many different viewpoints on 
how things work, and what should be done. Another thing that I learned is how difficult it is to 
collaborate with people on the other side of the world. I know that there is a time change, but it 
really became a reality when we tried to meet with them and that was quite an obstacle. The way 
that people think on the other side of the world is also very different than the way we think here 
which was interesting to think about. In seeing the way that they participated, and the way they did 
things it was neat to try to think from their perspectives. 

Similarly, a team member from Pix.ie stated: 

Working with international partners can be very stressful because of the language barrier. 

[However] I also learned allowing more minds to work on a project with a different background 
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betters the project. The international partners have a different point of view and contributed 
different ideas we had not thought of. 

Scores on the M-GUDS-s cognitive subscale (Relativistic Appreciation) did not change significantly from 
pre- to post-survey. This is similar to what others have reported (Strauss & Connerley, 2003); that is, changes on the 
M-GUDS-s cognitive and affective subscales tend to be smaller than those observed on the behavioral subscale. Still, 
students in this study agreed, at least “a little bit” that knowing how others differed from them “enhanced their 
friendships” or enabled them “to better understand their own problems.” (MGUDS-s survey items) Given the 
relatively short timeframe of this study, students had little opportunity to get to know, or to “become friends with,” 
their IPs. Thus, as indicated by the following two quotes, establishing “true” connections with the IPs was 
challenging: 

Collaborating with international partners to create a wiki chapter is both challenging and 
enlightening. The distance from our international partners caused various different obstacles when 
creating the wiki chapter. Both the language barrier and the methods of communication prevented 
us from establishing a true connection with our international partners (Team member, Pix.ie). 

Being online is easy to do with people of other countries because you can upload something and 
they can get it instantaneously. However, it is hard to get in touch with people in other countries 
because of the different time zones (Team member, Cellblock). 

In this study, students tended to focus on the task at hand; that is, interactions with their IPs were directed 
toward completing the project requirements, within the allotted timeframe. As such, students tended to view the 
primary role of the IPs as being that of helping them meet important project deadlines. Future research is needed to 
examine if cognitive changes are possible after engagement in lengthier or “friendlier” collaborations. 

Changes in Computer Skills, Perceived Comfort, and Perceived Knowledge for Using Web 2.0 Tools 

At the end of the course, students rated their computer skills, on average, at 3.98 (i.e., somewhat high) on 
the same 5-point scale used at the beginning of the course. This difference, from pre-post course, was significant at 
the .000 level (t = -12.46). Over 80% of the students rated their computer skills as somewhat or extremely high at 
the end of the course, compared to 40% at the beginning of the course. Post-course, none of the students rated their 
skills as being either extremely or somewhat low compared to 9.4 % at the beginning of the course. 

There were significant changes, from pre- to post-course, in students ’perceived comfort for engaging in 
international collaborations via Web 2.0 tools [t(201) = -8.62; p < .000], as well as in their perceived knowledge for 
using technology for teaching and learning [t(201) = -15.37; p < .000] (See Table 3). Students’ ratings of comfort 
increased from a mean of 3.62/6 (SD= 1 .02) to 4.37 (£D=68); ratings of perceived knowledge increased from a 
mean of 1.75/5 (5D=,56) to 2.58 (SD=. 65). 



Table 3. Pre- and Post-Survey Means and SDs on Perceptions of Computer Skills, Com fort, and Knowledge 



N=202 


Computer Skills 
(5 point scale) 


Perceived Comfort 
(6-point scale) 


Perceived Knowledge 
(5-point scale) 


Pre-survey Mean (SD) 


3.34 (.71) 


3.62 (1.02) 


1.75 (.56) 


Post-survey Mean (SD) 


3.98 (.61) 


4.37 (.68) 


2.58 (.65) 


Paired-t statistic 


-12.46 


-8.62 


-15.37 


p-value (2-tailed) 


.000 


.000 


.000 



These changes in students’ survey ratings are supported by comments made during the focus group 
interviews, as well as in students’ final project reflections. Two representative comments include: 



Web 2.0 tools will have a huge impact in my future classroom. 1 think it is extremely important to integrate 
technology. . . . technology can definitely enhance a student’s learning experience in ways teaching through 
a textbook or lecture can’t. ... I think having international experiences is also extremely important in my 
future classroom. Students should be aware of other nationalities and by the use of technology it may be 
easier for them to understand. Being able to show pictures of different countries from a tool like pix.ie 
would be a great experience (Team member, Pix.ie). 

I know I can and will use Jing in my future classroom as a means of teaching my students how to use 
different programs online or giving them examples of different lessons so they can review my lessons 
online. There are so many web 2.0 tools I know I can use eventually in preparation or in my classroom. 
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Some examples are blogging, file sharing, surveying tools. Not only can 1 use these tools for my students, 
my students can use them for their work and perhaps through the use of them, broaden their horizons to 
international experiences (Team member, Jing). 

Relationships among Variables 

To investigate the relationships among the defined research and demographic variables, we conducted a 
correlational analysis. Since it is fairly easy to achieve high correlation coefficients with larger samples, significance 
levels were set relatively high in order to discount high coefficients that were not meaningful. That is, we did not 
consider coefficients to be significant unless the probability of occurrence was less than p = .005. Thus, based on a 
critical r value ( df= 200) of .25, correlations between demographic variables and pre- and post-survey measures 
indicated no significant relationships among age, gender, or year in school and ratings of computer skills, cultural 
competencies, perceived comfort, or perceived knowledge. However, students’ pre-course ratings of their computer 
skills were significantly, but negatively, correlated with their pre-course ratings of perceived knowledge for using 
technology in their future classrooms (r = -.43). This means that students with greater levels of computer skills rated 
their knowledge for using technology in the classroom at lower levels than students who had lower levels of 
computer skills. Although it is unusual to see a negative correlation between these two variables, low, insignificant 
correlations have been reported previously (Ertmer et ah, 2003; Yildirim, 2000). One possible explanation for this 
finding is that students with fewer computers skills may have thought they could easily use technology in the 
classroom, once they gained more skills, while those with more substantial skills may have more readily recognized 
that, while they knew how to use technology for personal tasks, they didn’t know how to use those same skills to 
facilitate teaching and learning. 

There were no significant correlations between students’ perceived knowledge for technology use and the 
three subscales measuring cultural competency, pre or post. However, students’ perceived comfort for engaging in 
international collaborations using Web 2.0 tools was significantly correlated with cultural competencies (See Table 
4). More specifically, students’ pre-ratings of comfort significantly correlated with their pre-course ratings on the M- 
GUDS-s behavior subscale ( r = .72) as well as on the total score (r = .37), indicating a strong relationship between 
these variables. This is not unexpected given that on the Perceived Comfort survey items, students rated levels of 
comfort for using technology to engage in cross-cultural activities. Items on the M-GUDS-s behavior subscale were 
worded similarly, but without the technology component. Thus, one might expect students’ responses to be similar, 
particularly since students were generally comfortable using the Web 2.0 tools referred to in the items on the 
Perceived Comfort survey (Facebook, wikis, etc.). 

Post-ratings of comfort were also significantly correlated with post-ratings on all three MGUDS-s 
subscales: Relativistic Appreciation ( r = .32), Comfort with Differences {r = .40), and Diversity’ of Contact (r = .27), 
as well as on the total score (r = .41). That is, at the end of the project, students with higher ratings of comfort for 
using technology to engage in cross-cultural interactions also had higher perceptions of cultural competency than 
those students with lower ratings of perceived comfort. 

Finally, significant correlations among subscales of the M-GUDS-s and total M-GUDS-s score were 
obtained for both the pre- and post-surveys; however, because subscales contributed to the total score this is not 
particularly meaningful. Interestingly, however, pre and post M-GUDS-s scores were not significantly correlated 
with each other for any of the subscales or the total score. Although we saw a significant increase, from pre- to post, 
on two subscales and the total score, the lack of correlation between pre- and post-scores suggests that students’ 
ratings of cultural competency changed in different ways, perhaps due to their different levels of participation on the 
wiki project. 
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Table 4. Correlation Coefficients among Pre- and Post-Survey Measures (* p < .005) 





1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


1 

2 


-.02 




















3 


.05 


.03 


















4 


.11 


.32* 


-.08 
















5 


-.08 


-.02 


.53* 


-.08 














6 


.05 


.40* 


.01 


.59* 


-.06 












7 


.72* 


.05 


.11 


.10 


-.16 


.06 










8 


-.04 


.27* 


-.12 


.44* 


-.13 


.31* 


-.01 








9 


.37* 


.06 


.80* 


-.01 


.60* 


.03 


.54* 


-.11 






10 


.05 


.41* 


-.08 


.87* 


-.11 


.77* 


.06 


.74* 


-.04 





1: Pretest of Perceived Comfort, 2: Posttest of Perceived Comfort, 3: Pretest of M-GUDS 1 (Relativistic Appreciation; cognitive). 4: Posttest of 
M-GUDS 1, 5: Pretest of M-GUDS 2 (Comfort w/ Differences; affective), 6: Posttest of M-GUDS 2, 7: Pretest of M-GUDS 3 (Diversity of 
Contact; behavioral), 8: Posttest of M-GUDS 3, 9: Pretest of Total M-GUDS, 10: Posttest of Total M-GUDS 



After completing our correlational analysis, we used a series of hierarchical multiple regression analyses to 
examine the extent to which any or all of these variables could predict cultural competency in terms of the cognitive, 
affective, and behavioral components, outlined by Miville et al. (1999). Before the analyses, assumptions of 
normality, homoscedasticity, independence of errors, influential observations, and multicollinearity were examined 
and no violations were detected. Then, correlations were examined to establish the relationship between predictors 
and outcomes. Direct entry method was used to input predictors, using gap scores to control for pre-survey 
differences among students and to allow for a more focused emphasis on the development of students’ cultural 
competencies over the course of the project. For the second and the third analyses, perceived comfort was observed 
to have relatively higher correlation coefficients than the other variables. 

Table 5. Hierarchical Multiple Regression (HMR) Analysis Predicting Cultural Competencies in terms of Com fort 
with Differences (CwD), Diversity of Contact with Others (DoC), and Total M-GUDS-s Score (*p < .05) 



Models and predictors 


R 2 


AR 2 


AF 


sr 2 


B 


P 


HMR for Comfort with Differences (Affective) 
Model 1 


.005 


.005 


.48 








Gender 








.001 


.088 


.042 


Age 








.003 


-.015 


.058 


Model 2 


.023 


.018 


1.22 








Gender 








.0002 


.03 


.02 


Age 








.001 


-.01 


-.03 


Perceived Computer Skills 








.01 


-.13 


-.10 


Perceived Comfort 








.01 


.06 


.08 


Perceived Knowledge 








.001 


-.05 


-.04 


HMR for Diversity’ of Contact ( Behavioral ) 
Model 1 


.01 


.12 


1.19 








Gender 








.01 


.33 


.10 


Age 








.001 


.02 


.04 


Model 2 


.34 


.33 


32.26 








Gender 








.0002 


.05 


.02 


Age 








.004 


.03 


.07 


Perceived Computer Skills 








.00003 


.01 


.01 


Perceived Comfort 








.33 


.72 


.58* 


Perceived Knowledge 








.001 


.06 


.03 


HMR for Total 
Model 1 


.02 


.02 


2.05 








Gender 








.02 


.26 


.14 


Age 








.00004 


-.001 


-.01 


Model 2 


.11 


.09 


6.95 








Gender 








.007 


.16 


.09 
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Age 


.001 


.01 


.04 


Perceived Computer Skills 


.002 


-.06 


-.05 


Perceived Comfort 


.09 


.21 


.30* 


Perceived Knowledge 


.002 


-.06 


-.05 



Table 5 provides a summary of our findings from the three hierarchical regression analyses. With respect to 
Comfort with Differences, the M-GUDS-s affective subscale, the model including gender and age was not significant 
[F(2, 199) = .48 ,p> .05]. After controlling for gender and age, the variables of 1) perceived computer skills, 2) 
perceived comfort for engaging in international collaborations using Web 2.0 tools, and 3) perceived knowledge for 
using technology in future classrooms were included in the model. However, this model was not significant either 
[F(5, 196) = .93,/? >.05]. 

With respect to the Diversity of Contact behavioral subscale, the first step involving gender and age was 
not significant [F(2, 199) = 1.19, p > .05]. After controlling for gender and age, the variables of 1) perceived 
computer skills, 2) perceived comfort for engaging in international collaborations using Web 2.0 tools, and 3) 
perceived knowledge for using technology in future classrooms were included in the model. The model was 
significant [T 7 ^, 196) = 20.06, p < .001]. Together, all included variables predicted 34% of the scores on the 
Diversity of Contact subscale (i? 2 =.34). Among the regression coefficients of this model, perceived comfort for 
engaging in international collaborations using Web 2.0 tools contributed significantly to the model ( B (IC )= .72, tpc) 
(196) = 9.81,/? < .00 1 ) and explained 33% of variance (sr 2 (IC) =.33), while the contributions of the other variables 
were not significant. 

With respect to the M-GUDS-s total score for cultural competency, the first model including demographic 
variables was not significant. When other variables were added to the model, it was found to be significant. Similar 
to the results obtained for Diversity’ of Contact, perceived comfort for using Web 2.0 tools to engage in international 
collaborations was the only significant contributor (. B (IC )= .21, t (I Q (196) = 4Al,p< .001), and explained 9% of 
variance (sr 2 (IC ) =.09). 

Comparison of the three hierarchical regression analyses reveals that pre-service teachers’ comfort for 
engaging in Web 2.0-based collaborations is a good predictor of cultural competencies, especially behavioral aspects, 
as measured by the Diversity of Contact subscale. Thus, the findings from this study suggest that participation in a 
cross-cultural wiki project, which enabled interactions among both local and international peers, was an effective 
strategy for increasing pre-service teachers’ cross-cultural awareness and acceptance of differences among others. 
From pre- to post-project, there was a general increase in students’ attitudes of openness and acceptance of the 
various cultures to which team members and international partners belonged (Miville et al., 1999). 

Discussion 

Findings from this study demonstrated that engaging in a five-week cross-cultural wiki-development 
project had a significant impact on the development of students’ cultural competencies, particularly in terms of their 
behavioral and affective competencies. This is a promising result as cultural competencies are generally difficult to 
change (Nunez, 2000), especially in a short period of time. While previous work has documented, qualitatively, the 
impact of immersion experiences on cultural competency (Dekaney, 2008; Lee, 2009), this study was able to show 
changes, both quantitatively and qualitatively, after engagement in a relatively short-term project. This suggests that 
integrated course experiences such as the one described here have the potential to substitute for longer, more 
expensive programs, at least in terms of increasing students’ awareness of and appreciation for diverse others. At the 
very least, wiki-based collaborations may offer a reasonable pre-cursor or follow-up to a more intensive experience. 

Findings from this study also indicated that participation in a cross-cultural wiki-based collaboration had a 
significant positive impact on students’ perceived comfort for using Web 2.0 tools to collaborate with diverse others 
around the world. Given that students of this generation are generally comfortable using Web 2.0 tools for the 
purposes of social networking (Project Tomorrow, 2010), it appeared relatively easy for them to envision being able 
to use these same tools for networking with an extended group of “friends.” In addition, students’ perceived 
knowledge for using Web 2.0 tools for teaching and learning increased significantly from pre -post project. 

According to Lawless and Pellegrino (2007), “Technological literacy has fast become one of the basic skills of 
teaching” (p. 580). Although most pre-service teachers graduating today are likely to be “digital natives” (i.e., 
comfortable using a variety of technology tools), their knowledge of how to use these tools to support teaching and 
learning tends to be underdeveloped (Lei, 2009). Due to the nature of this project, in which students were asked, 
specifically, to consider how their assigned Web 2.0 tools could be used for teaching and learning, students became 
aware of the potential pedagogical uses of the tools. Moreover, through the final showcase event, students gained an 
understanding of the great variety and number of tools available. 
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Finally, students’ perceived comfort for participating in Web 2.0-based international collaborations was a 
relatively strong predictor of changes in cultural competency, explaining 33% of the variance in the behavioral 
subscale. Diversity’ of Contact, and 9% of the variance on the total M-GUDS-s scale. This suggests that as students 
become more comfortable using technology to engage in virtual collaborations, it may not be too much of a stretch 
to consider engaging in other types of cross-cultural activities as well. Previous research (Cone, 2009; Maleski, 
Phillion, & Lehman, 2005) has reported similar findings: once students have completed initial experiences in cross- 
cultural settings, they become more open to participating in other experiences. For example, Cone found that pre- 
service elementary teachers who participated in community-based service-learning in culturally diverse communities 
were more likely to “change their feelings about, diverse student groups before they enter[ed] their student teaching 
semesters” (p. 28). Similarly, Maleski et al. (2005) described how pre-service teachers, through weekly 
videoconferencing sessions with low-income, minority students, changed their ideas about where they wanted to 
teach, resulting in a greater openness to working in urban settings, which served diverse groups of students. 

It is important to note that although the regression analyses accounted for a moderate amount of the 
possible variance, an even greater portion was not explained. What additional factors may need to be considered? 

For example, it is fairly clear that students in this study did not participate equally in the wiki collaboration 
(especially given their different roles and responsibilities) and that some teams had more robust interactions with 
their international partners than others. It is possible that teams who interacted with IPs whose first language was 
English experienced more immediate benefits than those who interacted with IPs from Russia, South Korea, or 
Sweden. Furthermore, we did not gather demographic data about our students’ cultural or ethnic backgrounds or 
information about their previous cross-cultural experiences. It is quite possible that these additional factors 
influenced students’ individual responses to the international, wiki-based collaboration and, ultimately, to their 
perceived willingness to go beyond the project boundaries to engage in additional cross-cultural activities. 

Limitations and Suggestions for Future Research 

Participants in this study were pre-service teachers, enrolled in a required introductory educational 
technology course. As such, generalizability is limited to similar students enrolled in similar courses. Additionally, 
the project occurred over a five-week time period, which may have limited the types of changes we were able to 
observe. Given that students didn’t get to know their international partners on a personal basis, we saw little change 
in the value they assigned to these types of activities, at least as measured by the cognitive subscale of the M- 
GUDS-s. Future research is needed to examine changes in other groups of participants who engage in integrated 
course experiences, especially those involving different time frames and different course structures. Future efforts 
might also be directed toward providing increased opportunities or more explicit encouragement to students to 
interact with their IPs on a more personal level. Finally, we didn’t examine the contributions of individual team 
members to the wiki chapters; it is possible that changes in students’ post-survey scores were influenced by factors 
we did not measure, such as previous knowledge, skills, and/or experiences with diverse others and/or levels of 
participation on the wiki. It would be useful to gather this data in future studies to determine how these factors 
impact initial and changing cultural perceptions. 

Implications 

The results of this study have implications for colleges and universities tasked with preparing their students 
for a knowledge-based global economy. Given that the participants in this study were enrolled in a required course 
that engaged them in a short-term cross-cultural experience, observed changes in cultural competency are 
particularly promising. Previous studies (Causey, Thomas & Armento, 2000; Pence & Macgillivray, 2008) have 
demonstrated the difficulties involved in changing cultural competencies through voluntary, short-term experiences, 
including those with an immersion component (e.g., study-abroad). Thus, a wiki-based collaboration may offer 
instructors the opportunity to increase their students’ appreciation for other cultures with fewer issues than a full 
immersion program involves (Bellamy, 2006; Willard-Holt, 2001). With careful planning and cooperative 
international relationships, Web 2.0 technologies have the potential to prepare our current students for the diverse 
populations they are likely to face in their future classrooms (Schoorman, 2002; Maleski et al., 2005). Finally, 
because the participants of this study were pre-service teachers, there are additional implications for their future 
students. That is, students in this study indicated a strong desire to incorporate international activities within their 
future classrooms using the tools they learned about in this project. 

Conclusion 

Regardless of discipline, today’s employers are seeking graduates who possess skills and knowledge that go 
beyond basic technical proficiencies. According to the Partnership for 21 st Century Skills (2007), the workforce of 
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the 21 st century must be digitally literate and globally competent, possessing effective communication, collaboration, 
and problem-solving skills. This is particularly true of our future teachers, who will prepare our children for a 
workplace that depends on these skills (U.S. Department of Education, 2004). As universities heed the call to expand 
pathways to global education, integrative course experiences offer a cost-effective alternative to traditional study 
abroad programs. Particularly in the current economy, infusing international experiences into on-campus courses 
offers a means by which every student can participate. According to Larusson and Alterman (2009), “using 
collaborative technology [such as wikis] to extend the physical borders of the classroom can be of significant value” 
(p. 397). 

As noted by Brown (2006) and his colleagues (Brown & Adler, 2008), the emergence of Web 2.0 tools has 
generated enormous possibilities for international collaborations by shifting attention from access to information to a 
ccess to other people. Wiki technology, in particular, provides students with collaborative opportunities that, in the 
long run, can initiate them into a knowledge creating culture and enable them to see themselves as part of a global 
effort to advance knowledge (Huijser, Bedford, & Bull, 2008; Parker & Chao, 2007; Pfeil Zaphiris, & Ang, 2006). In 
this study, students’ wiki experiences resulted in an expanded view of working with diverse others; that is, at the end 
of this project team members understood and valued input from others, both locally and at a distance, who were 
from their own, as well as other, cultures. Students gained a greater appreciation and respect for the differences of 
others and other cultures. In addition, the international collaboration using Web 2.0 technologies created a greater 
awareness of the benefits (easiness, usefulness, and importance) that various Web 2.0 technologies can play in their 
future classrooms. 
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Abstract 

The purpose of this study was to document the design decisions and challenges encountered by designers of 
an AWBLE which was developed to provide a platform for tailoring instructional strategies, content, resources, and 
interfaces based on two learner characteristics —motivation and prior knowledge. 
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Introduction and Background 



Adaptive Web Based Learning Environments 

Adaptive Web-Based Learning Environments (AWBLEs) are one form of adaptive instruction which 
attempt to address the issue of individual differences. AWBLEs do so by providing mechanisms for individualizing 
instruction (e.g., content, interface, and strategies) and providing users with a more personal experience by 
incorporating customized instructional strategies, resources, assessments, and interfaces (Abidi, 2009; Inan & 

Grant, 2008). As a general outline, AWBLEs (1) gather learner information; (2) build a learner user model based on 
that information; (3) make inferences based on the collected data and employs adaptive methods to accommodate 
each individual based on their developed user model; and (4) continuously monitor user actions and interactions in 
order to update the user model based on newly collected data. 

Although it is quite easy to comprehend what AWBLEs are and what they do; the design and development 
process of AWBLEs, however, is a challenging and complicated topic. Adding to the challenges and complexities is 
the fact that current research is very limited in respect to documenting the decisions that AWBLE designers make in 
order to design their adaptive systems. These decisions can be grouped into three types: user related, system 
related, and user to system related decisions. 

Purpose of the Study 

The purpose of this study was to document the decisions and challenges encountered by designers of an 
AWBLE which was designed to provide a platform for tailoring instructional strategies, content, resources, and 
interfaces based on two learner characteristics —motivation and prior knowledge. The researchers of this study 
expect that their documentation will help instructional designers develop better adaptive web-based instruction 
which more closely meets the instructional needs of students. 

User Related Decisions 

User related decisions encompass decisions on which learner characteristics to adapt for, how learner 
characteristics will be measured; and how measurements will be used to define learner user models. 

Learner Characteristics 

Although researchers agree on the influential effects of individual differences during learning, the question 
still remains as to which learner characteristics should be considered when designing AWBLEs (Scheiter, Gerjets, 
Vollmann, & Catrambone, 2009; Shute & Zapata-Rivera, 2007). In past research, variables have included prior 
knowledge (Brusilovsky, 2003), cognitive/learning style (Akdemir & Koszalka, 2008), goals (Brusilovsky, 1998), 
beliefs and attitudes (Hudlicka & McNeese, 2002), motivation (Far & Hashimoto, 2000), language (Carro, 2002; 
Espada, 2006), gender (Milne, Cook, Shiu, & McFadyen, 1997), and disabilities (Fink et al., 1996, 1998). 

Due to the complexity of adapting for multiple learner characteristics simultaneously, the designers of this 
study limited adaptation to two learner characteristics, namely student prior knowledge and motivation. Many 
studies have considered prior knowledge (Brusilovsky, 2003; Chen & Paul, 2003) and motivation (Far & 

Hashimoto, 2000; Johns & Woolf, 2006) to be two of the most important factors which should influence the design 
of web-based instruction. 

Learner Characteristics Measurements and Learner User Models 

In order to make inferences about the learner, researchers must determine how learner characteristics will 
be measured and how measurements will be used to cluster students into learner user models. For the assessment of 
prior knowledge, designers of this AWBLE used a locally developed 10 item multiple choice instrument. Items 
measured prior knowledge for each section of the tutorial. For motivation towards the material, the researchers 
adapted items from Keller’s Instructional Material Motivational Survey (IMMS) instrument. This instalment is 
based on the Attention, Relevance, Confidence and Satisfaction (ARCS) motivational design model (Keller, 1987a, 
b) and is widely applied to the motivational evaluation of instructional materials (Chang & Lehman, 2002; Means, 
1997.) 
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As with the number of learner characteristics, the researchers of this study limited adaptation to two 
discrete levels— high and low. Combinations of these high/low prior knowledge and motivation levels served to 
cluster students into four learner user models: (1) low motivation and low prior knowledge, (2) low motivation and 
high prior knowledge, (3) high motivation and low prior knowledge, and (4) high motivation and high prior 
knowledge. 



System Related Decisions 

System related decisions are concerned with what predefined user models the system will have, which 
instructional strategies will be used, what content will be taught, and how content will be designed for each system 
user model. 



Predefined System User Models 

Because content and resources cannot design, develop, and customize themselves to users while they interact with 
the system, AWBLEs must have predefined system user models. Each system user model should correspond to one 
of the possible learner user models. Having such system user models allows designers to customize content for the 
learners prior to instruction. For the current adaptive system, four system user models were developed corresponding 

to one of the four learner user models. 

Instructional Strategies for Predefined System User Models 

Instructional strategies affect student learning differently depending upon their structural characteristics 
(Jonassen & Grabowski, 1993; Scheiter et al., 2009; Tobias, 1987). Therefore, it is essential for AWBLE designers 
to identify relevant instructional strategies, so that instruction can be adapted effectively (Azevedo et al., 2008; 
Carro, Pulido, & Rodriguez, 1999; Clarke, Ayres, & Sweller, 2005). For this study, researchers reviewed the 
literature for instructional strategies relating to motivation and prior knowledge. A list of sample instructional 
strategies, grouped by adaptive method, can be found Table 1. The symbol “X“ signifies that those strategies were 
emphasized for the respective learner characteristic. Having such a table allowed designers to quickly retrieve and 
incorporate relevant strategies into the design of instruction for each system user model. 



Table 1 Instructional Strategies for Adaptive Instruction (Abbreviated) 



Learner Characteristics* 


Strategies 


LK 


H 

K 


LM HM 


Adaptive content presentation 


1 . 


Minimal coherent explanations (McNamara et al., 1996) 




X 




2. 


Additional explanations, resources, and content (McNamara et al., 1996; 
Kalyuga et al. 1998 ) 


X 






3. 


Use of graphics /animation (Mayer & Gallini 1990; Mayer et al. ,1995; Song 
& Keller, 2001) 


X 




X 


Adaptive Navigation/Control 


1 . 


Help the user to understand the structure of the overall hyperspace and his or 
her absolute position in it. (Chrysostomou, Chen, & Xiaohui, 2009; Shapiro, 
1999) 


X 






2. 


Rich linking technologies such as adaptive annotations and multiple link 
generations (Shin et al., 1994; Brusilovsky et al., 2009 ) 




X 


X 


Adaptive Instructional Activities 


1 . 


Activate prior knowledge (eg.. Knowledge maps, advance organizers) 
(Lambiotte & Dansereau, 1992; Song & Keller, 2001) 




X 




2. 


Emphasize key/important points (e.g., illustration or visual) (Song & Keller, 






X 
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2001) 


3. More elaboration (Tobias, 1976; Kalyuga et al. 1998) 


X 




Adaptive Assessment, Support, and Feedback 


1. Minimal guidance or no guided practice (Tuovinen & Sweller, 1999) 


X 




2. Well Guided & low paced. Guided practice before “on your own practice” 
(Tuovinen & Sweller, 1999) 


X 




3. Question response feedback within content (Song & Keller, 2001) 


X 




Adaptive Interface 


1. Use a consistent screen format (Song & Keller, 2001) 


X 


X 


2. Intermingle information presentation screens with interactive screens (Song 
& Keller, 2001) 


X 


X 


3. Use attention getters (e.g., inverse and flash in text and patterns in pictures) 
(Song & Keller, 2001) 


X 







*LK= Low Prior knowledge; HK= High Prior Knowledge; LM= Low Motivation; HM= High Motivation 



Content Topic and Design 

Before incorporating instructional strategies into the design, AWBLE designers must first decide what 
specific content topics will be presented. For this AWBLE tutorial, content topics came from an undergraduate 
statistics course where basic introductory statistics topics are introduced. These topics, separated into three sections, 
were: 

• Section 1: Basic Theory on Probability (random phenomena, law of large numbers, etc) 

• Section 2: Terminology of Sample Space (event, sample spaces) 

• Section 3: Basic probability rules (Probability rules, probability of event) 

For each section’ page, the design of content varied dependent on which of the four predefined system user 
models the material was prepared for. For example, assets from pages for system user model 1 (low motivation/low 
prior knowledge) contained additional worked examples and incorporated attention getting resources, such as Flash 
guided practices with animated interactions, colorful graphics, etc. 

User to System Related Decisions: 

After user related and system related decisions have been made, designers must then make user to system 
related decisions. These decisions are concerned with how the system will match learner user models to system user 
models; and how the system will monitor user actions and update learner user models. 

Matching and Updating Student’ Predefined User Models 

For the current system, students began by logging into the system. Once they were logged in, students 
were required to fill out a knowledge pretest which measured their prior knowledge, and a motivation survey which 
measured their initial motivation. Dependent on the results of these assessments, students were placed into one of 
four learner user models and given a composite factor value (CFV) for each section of the tutorial. Using respective 
CFVs for each section and sql statements, learners were then matched to the content which had been prepared for 
predefined system user models having the same CFV. For example, students belonging to learner user model 1 (low 
motivation/low knowledge) would be matched with system predefined user model 1 and then be presented with all 
the assets that incorporated relevant strategies, resources, and examples customized for students belonging to that 
learner user model. For each tutorial section, it was possible for students to be placed into different learner user 
models dependent on their assessed motivation and prior knowledge levels for the topics of that respective section. 
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Discussions 



In order to design an effective AWBLE there are many decisions that must be made prior to development. 
These decisions include user related decisions, system related decisions, and user to system related decisions. When 
making these decision designers encounter many challenges. Below are some of the design challenges which the 
researchers of this study encountered. 

Adaptive Strategies Challenges 

Regarding adaptive strategies, the critical limitation came from the inadequacy of previous research 
regarding the implementation of adaptive strategies (Shute & Zapata-Rivera, 2007; Tsandilas & Schraefel, 2004). 
There continues to be inconsistencies regarding the matching of strategy with student characteristics (Federico, 

1999; Lee & Park, 2007; Scheiter & Gerjets, 2007). For this study, researchers implemented research based 
instructional design strategies outlined in Table 3; however, it continued to remain unclear as to how strategies 
should be combined and/or which strategies should be prioritized with the two given factors. This challenge was a 
major issue because it had a direct impact on the design of the AWBLE. 

Student Assessment Challenges 

Developing and implementing assessment is challenging for adaptive instruction (Shute & Zapata-Rivera, 
2007; Weibelzahl, 2005). In addition to developing instruments specific to the instruction, there are significant 
concerns regarding the validity and reliability of instruments to measure all learner characteristics— especially 
student attitudes and motivation. Compounding this issue, was the limited prior research regarding the most 
efficient frequency of student assessment (Burgos, Tattersall, & Koper, 2007); and how assessment measures can be 
used to define learner user models and matched with instructional strategies. 

Conclusion 

AWBLEs provide many advantages to students because they more closely address the issue of individual 
differences. If implemented correctly AWBLEs can increase student performance, motivation, and attitudes, while at 
the same time decrease learning time and usability problems (Brusilovsky, Sosnovsky, & Yudelson, 2009, Dogan, 
2008; Tsianos et al., 2009). Flowever, before reaping these benefits, there are many design decisions and challenges 
that AWBLE designers must address. 
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Enhancing the Connectivity and Sustainability in Asynchronous Online Discussion 

Abstract 

Lack of connectivity and sustainability are two common problems in asynchronous online discussion. In 
this paper, the author proposes that the two problems may have to do with the structure of threaded forums, that is, 
the design of threaded forums may constrain participants from having well-connected and sustained discussions. A 
new discussion environment was developed to promote the connectivity and sustainability of asynchronous online 
discussion, and an exploratory study was conducted to test the effectiveness of the new environment. 



Computer-mediated communication is becoming an increasingly important mechanism for supporting 
student learning (Anderson & Kanuka, 1997; Duncan-Howell, 2010; Paulus & Roberts, 2006). In particular, 
asynchronous online discussion provide support for community building and productive discourse (DeWert, 
Babinski, & Jones, 2006; Joeng, 2003). It allows learners to go beyond the space and time constraints, 
communicating with and learning from each other (McCrory, Putnam, & Jansen, 2008). 

Researchers, however, have found it hard to establish productive interaction in asynchronous online 
discussion. One of the problems is lack of connectivity. Connectivity reflects how well participants connect or 
respond to each others’ ideas. Studies show that participants do not make frequent connections to each others' ideas. 
Instead, they post condensed expositions of their own ideas, without attending or responding to the ideas of others 
(Hara, Bonk, & Angeli, 2000; Larson & Keiper, 2002). As a result, there is little negotiation of meanings and 
building up of ideas. Another problem is lack of sustainability. Sustainability reflects how well the discussion is 
maintained and developed. A high level of sustainability has been regarded as an important characteristic of 
productive discussion (Guzdial & Turns, 2000; Hewitt, 2005). Online discussions, however, are often brief, and the 
limited growth and development of discussions prevents learners from effectively constructing shared knowledge 
(Guzdial, 1997; Hewitt & Teplovs, 1999). In this paper, the author attempts to design a new environment that better 
supports the two qualities of online discussion. 

Designing a New Environment 

The assumption is that participants are more likely to make connections and sustain conversations if the 
connectivity and sustainability of discussion are made easier to perceive. Two design principles based on the 
assumption are: (a) the connections and relations among posts should be made more visible, and (b) the 
development and continuity of the discussions should be made more visible. 

A new environment called discussion map was developed in Mindomo (www.mindomo.com), an online 
collaborative concept map site (see Figure 1). In this environment, the discussion question is presented in a bubble at 
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the center. When participants log in, they can respond to the discussion question and existing posts by adding sub- 
bubbles. A note can be attached to each bubble, allowing participants to write more elaborated posts. One can read 
the posts by clicking the note icons. The discussion map looks similar to concept maps, but is different from concept 
maps in that (a) a bubble in discussion map represents a post rather than a concept, and (b) a link in discussion map 
only represents one type of relationship - a reply to a previous post. 




In the discussion map, the connections and relations between posts are presented visually, making it easy to 
perceive. When a response connects to more than one post of different threads, green lines can be used as cross-links 
to indicate the connections. In addition, in the discussion map, the growth and development of threads and sub- 
threads can be easily seen. Participants can instantly identify a thread or sub-thread, focus on its development and 
further develop it. 

To understand the effectiveness of the new environment, an exploratory study was conducted by comparing 
discussions in the threaded forum with those in the discussion map. The research questions are: 

1 . How did the discussions in the discussion map differ from those in the threaded forum in terms of 

connectivity? 

2. How did the discussions in the discussion map differ from those in the threaded forum in terms of 

sustainability? 

3. How did students perceive the discussions and learning in the two environments? 

Method 



Participants 

Participants were six males and 10 females enrolled in an online master’s level education course. Most (14 
out of 16) were practicing K-12 teachers. Among them, 10 (62.5%) were within the age range of 26 to 35, one 
(6.25%) was below 25, and five (31.25%) were beyond 36. All but three had taken at least one online course prior to 
this one. 

Procedures 

The online course was delivered via ANGEL course management system. It was structured as a series of 
learning units, with each unit including readings, online activities, and small-group discussions on a topic related to 
teaching and learning with technology. There were altogether four discussion groups. Each group consisted of four 
students. Prior to the study, written tutorial was provided to the students on how to have discussion in the new 
environment. In addition, all students had two one-week long discussions in the discussion map to get familiar with 
the environment. During the study, two topics were selected for discussion, and each discussion last one week. The 
first discussion topic concerned the educational value of simulation. Students explored and evaluated a few 
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simulation sites, and discussed the strengths and weaknesses of simulation. The second topic had to do with 3D 
virtual worlds, such as Second Life. Students discussed the strengths and weaknesses of Second Life as a potential 
learning experience. In discussing topic one, students in Group 1 and 2 had discussion in the Mindomo discussion 
map; students in Group 3 and 4 had discussion in the ANGEL threaded forum. In discussing topic two, discussion 
environments were reversed, with students in Group 1 and 2 using the ANGEL threaded forum and students in 
Group 3 and 4 using the Mindomo discussion map. At the end of the two weeks of discussions, students completed a 
survey focusing on their experiences and perceptions of participating and learning within the two environments. 

Data Analysis 

Online discussion posts. Students’ posts in the two environments were analyzed using Kear’s (2001) 
methods to uncover the degree of connectivity and sustainability in the discussions. To measure connectivity, the 
number of system links and semantic connections were counted. System links were created by students when they 
used the “reply” function in the threaded forum or created a physical link in the discussion map. To determine 
whether the system links truly represented semantic connections between posts, a semantic connection was noted if 
a post was clearly a reply to a previous post, or a continuation of an existing discussion. The number of within- 
thread and cross-thread links/connections in the discussions was also counted. To measure sustainability, the 
following things were recorded: (a) the number of isolated (i.e. unthreaded) posts and (b) the number of posts per 
thread. 

Survey. The survey contained a set of Likert-questions asking students to rate their experiences and several 
open-ended questions asking them to explain their ratings. Sixteen students took the survey. Their responses to 
Likert-questions were coded into scores ranging from -2 to 2 based on the degree to which they agreed with the 
statements, where -2 = “strongly disagree”, and 2 = “strongly agree”. The data were analyzed using MANOVA. 
When an omnibus difference was found, follow-up ANOVAs were conducted. Student responses to open-ended 
questions were used to support the findings. 



Results 



Connectivity 

Examination of the discussions suggested that all the system links in the two environments were 
semantically meaningful, that is, students created these system links only when they directly replied to a previous 
post or previous posts. The mean and standard deviation of system links and semantic connections in the two 
environments were presented in Table 1. What is worth noting is that, in the threaded forum, there was no cross- 
thread system link, because when students use the “reply” function in the threaded forum, they can only create a 
system link to one post rather than multiple posts. Two posts in the threaded forum, however, were coded as cross- 
thread semantic connections because they made connections to posts in multiple threads by explicitly mentioning 
those posts. 

Table 1. Mean and Standard Deviation of Links/Connections Created by Individual Student in the Two 
Environments (n=16) 





Within-Thread Connection* 


Cross-Thread Connection F P 




Forum 


Map 


Forum 


Map 


System 


2.63 (1.75) 


4.75 (3.32) 


0.00 (0.00) 


0.31 (0.60) 


Semantic 


2.63 (1.75) 


4.75 (3.32) 


0.13 (0.34) 


0.31 (0.60) 3.77 .04 



* p < .05 



A MANOVA analysis of the number of within- and cross-thread semantic connections revealed that there 
was an overall difference between the two environments in terms of semantic connections (See Table 1). Follow-up 
ANOVAs suggested that the number of within-thread connections was significantly different between the two 
environments [F (1, 30) = 5.70, p < .05], but the difference in the number of cross-thread connections was not 
significant [i r (l, 30) = 1.17 , p > .05]. 

Sustainability 

Over the two weeks of discussions, students produced 61 posts in the threaded forum with four isolated 
posts, and 97 posts in the discussion map with five isolated posts. On average, there were 4.41 posts per thread in the 
discussion map, and 2.65 posts per thread in the threaded forum (See Table 2). The ANOVA analysis suggested that 
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threads in the discussion map contained significantly more posts than those in the threaded forum [F (1, 43) = 4.63, 
p < .05]. 

Table 2. Mean and Standard Deviation of Posts Per Thread in the Two Environments (n=45) 





Mean (SD) 


F 


P 


Threaded Forum 


2.65 (1.37) 


4.63 


.04 


Discussion Map 


4.41 (3.66) 







Student Perceptions 

Results obtained from analyzing student survey responses are presented in Table 3. There is a statistical 
difference in terms of student perceptions of the two environments [F (1, 15) = 12.40, p < .05]. More specifically, 
students reported that, compared to the threaded forum, the discussion map has made it easier to see (a) how a post 
is connected to others [^(1, 15) = 9.29, p <.01] and (b) the growth and development of a thread [/’(l, 15) = 10.91 ,p 

< .01]. In addition, they thought it easier to make connections [F (1, 15) = 60.28, p < .001] and extend a thread [F (1, 
15) = 10.21, p < .01] in the discussion map than in the threaded forum. But they did not rate the discussion map as 
significantly better to support sustained discussion than the threaded forum [F (1, 1 5) = 3.38, p > .05]. 

Students also reported that they were more engaged [F (1, 15) = 11.67,/? < .01], and had more fin [F (1, 15) 
= 30.77,/? < .001] when having discussion in the discussion map as compared to in the threaded forum. But they did 
not feel that they participated more actively in one environment over the other [F (1, 15) = 3.30,/? > .05]. Finally, 
students found no difference in terms of the level of productivity of discussion [F (1, 15) = 4.36, p > .05], but they 
perceived that the discussion map helped them learn the content better than the threaded forum [F (1, 15) = 7.64,/? 

< .05], 

Table 3. Mean and Standard Deviation of Student Ratings for the Two Environments (n=16) 



Survey Items 


Threaded 

Forum 


Discussion 

Map 


F 


P 


1. Connectivity 

Easy to see how a post is connected to others * 


.25 (1.06) 


1.44 ( .96) 


9.29 


.008 


Easy to build within-thread and cross-thread 
connections ** 


-.38 (.89) 


1.69 (.48) 


60.28 


.000 


Easy to respond to an existing post from 
multiple directions ** 


-.31 (.87) 


1.31 ( .87) 


21.30 


.000 


Easy to synthesize ideas across the posts * 
2. Sustainability’ 


.25 (1.13) 


1.25 ( .77) 


10.91 


.005 


Easy to see the growth and development of a 
thread of discussion* 


.38 (.89) 


1.38 ( .81) 


10.91 


.005 


Easy to extend and build ideas on a thread of 
discussion * 


.25 ( .93) 


1.38 ( .89) 


10.21 


.006 


Easy to have sustained discussion on a thread 
of discussion 


.38 (.81) 


1.00 (.97) 


3.38 


.086 


3. Engagement 
I participated actively 


.81 ( .66) 


1.25 ( .77) 


3.30 


.089 


I was deeply engaged * 


.25 ( .77) 


1.13 ( .81) 


11.67 


.004 


I had fun ** 


- .06 ( .68) 


1.25 ( .58) 


30.77 


.000 


4. Perceived Learning 
The discussion was productive 


.88 (.62) 


1.25 ( .58) 


4.36 


.054 


The discussion helped me learn the content * 


.69 ( .79) 


1.25 ( .68) 


7.64 


.014 



* p < .05 

**p < .001 
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Discussion 



Kear (2001) argued that the changes in the way discussion threads are visually displayed could affect the 
how participants interact during the discussion. This study provided further evidence for the statement. The 
discussion map developed in this study is different from the threaded forum in that the discussion map did a better 
job in visually displaying the interconnections of posts and development of threads in discussion. Results of the 
exploratory study showed that discussions in the two environments were different in a few ways. 

First, students made more connections when had discussion in the discussion map than in the threaded 
forum (4.41 versus 2.65 on average). This finding is consistent with student responses in the survey, where they 
identified the discussion map as a better environment for making connections than the threaded forum. One possible 
explanation is that, as compared to the threaded forum, the discussion map has made it easier to see how posts are 
connected with each other. Another reason could be that it is simply easier to make a connection, especially a cross- 
thread connection, in the discussion map than in the threaded forum. As one student wrote, “I guess I'm not sure 
how you build a cross-thread connection on threaded forums, except by mentioning it in your post. In the new 
environment, you can connect with graphic arrows, very cool.” Interestingly, the analysis of discussion posts 
suggested that students did not make significantly more cross-thread connections in the discussion map than in the 
threaded forum (a total of five posts in the discussion map versus two posts in the threaded forum). Perhaps, making 
cross-thread connections requires more than a supportive environment. It also requires participants to carefully 
examine the themes and details of different threads and intentionally look for the connections. 

Second, the discussions in the discussion map were more sustained than those in the threaded forum. A 
thread in the discussion map contains more posts than that in the threaded forum (4.41posts per thread in the 
discussion map, and 2.65 posts per thread in the threaded forum). Students agreed that the discussion map was better 
in visually presenting the growth and development of discussion than the threaded forum. To use their words, “[In 
the discussion map,] you could see how many people responded to a post and how it connected to other posts at the 
same time ” and “the visual concept mapping feature naturally lends itself to making connections, developing, and 
maintaining an atmosphere of conversation”. 

Based on the survey responses, students thought it easier to “extend and build ideas” in the discussion 
map, but they did not believe that the discussion map were significantly better than the threaded forum in supporting 
sustained discussion. The relatively low ratings of the two environments in terms of sustainability (0.38 for the 
threaded forum and 1.00 for the discussion map) suggested that students might feel discussions in neither 
environment had reached the optimal level of sustainability. It is worth noting that, in addition to the discussion 
environment, many other factors can affect the sustainability of a discussion. For example, “the absence of 
substantial differences of opinion on most of the issues” (p. 15) or “having the deadlines so close to the end of the 
discussion” (p. 21) may make the discussion less sustained (Bullen, 1998). In this study, one week might not be long 
enough for discussions to be fully developed. When designing such a study in the future, researchers should take 
other factors including the duration of discussion, the characteristics of participants, group size, and discussion topic 
into consideration. 

Third, students felt that it was more fun and engaging when having discussion in the discussion map than 
in the threaded forum. When asked why, students had different answers. Some students explained that the visual 
display had made the discussion map more fun - “The visual set-up of Mindomo [discussion map] was more 
appealing and fun to work with. It also gave more opportunities to think in many directions because of the web like 
nature of it.” Some other students thought that Mindomo allowed them to add little symbols such as question mark 
icons and thumb up icons to the discussion, which made discussion more fun. 

Students perceived that they learned the content better through the discussions in the discussion map than 
in the threaded forum. Student perceived learning, however, may not truthfully reflect their actual learning. One 
student wrote that “being more excited to use Mindimo [discussion map], I feel I got more out of the discussions and 
they were more productive to me. I learned the views of my group on ANGEL [threaded forum] too, but I wasn't as 
excited and interested in it and therefore feel that I got less out of it”. This suggested that some students felt that they 
learned better because they were more excited or more interested when using the discussion map. That is, their level 
of engagement affects how they perceived their learning. The actual learning outcomes were not measured in this 
exploratory study. Therefore, it remains unknown whether the discussions in the two environments made a 
difference in student learning. 
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Conclusion 



Garrison and Cleveland-Innes (2005) pointed out that online discussion must be designed and structured 
in a way that facilitates clear discussion threads, avoids disjointed monologues, and moves the discussion to higher 
levels of thinking. This paper is a contribution towards this goal. It presents the rationale for developing a new 
environment that has a potential to enhance connectivity and sustainability in online discussions, and reports the 
results of an exploratory study. Due to the small sample size, the extent to which the results can be generalized is 
limited. All the findings should be interpreted with caution. The study, however, is informative and has meaningful 
implications for future research. 
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The Effects of Two Factors on Asynchronous Discussions in a Chinese Online 
Course: Types of Discussion Activities and Discussion Leaders 



Ninghua Han 
Jongpil Cheon 
Sungwon Chung 
Texas Tech University 

Abstract 

This study examined the effects of two factors on asynchronous discussions in a Chinese online course. 

One factor was different types of online discussion activities (problem-solving, sharing information, case study, and 
debating). Another factor was different discussion leaders (instructor-lead, tutor-lead, peer-lead, and no leaders). 

The findings indicated that debating discussion activity could facilitate students to achieve higher level of 
knowledge construction than sharing information activity. However, different discussion leaders showed no 
significant effects on students’ meaningful participations. 

Introduction 

Online discussion is a main way of communication between teachers and students in distance education. 
Previous studies have proposed many instructional interventions that could influence the effects of online discussion, 
such as discussion group effect size (e.g., Du, Zhang, Olinzock, & Adams, 2008), discussion facilitation approaches 
(e.g., Hew & Cheung, 2008; Schellens & Valcke, 2006), discussion group structure (e.g., Jeong, 2004; Rose, 2004), 
and online students’ characteristics (e.g., Krentler & Willis-Flurry, 2005; Spatariu, Quinn, & Hartley, 2007). For 
example, Kanuka, Rourke, & Laflamme (2006) applied five different instructional strategies to online discussion: (a) 
WebQuest, (b) debate activity, (c) nominal group, (d) reflective inquiry, and (e) invited expert activity. Their results 
showed that the WebQuest and debate group had higher participation and cognitive presence than other groups. 
Regarding group leaders, on the other hand, tutor-supported discussion yielded significantly higher levels of 
knowledge construction as compared to students in the role-supported group (Wever, Keer, Schellens, & Valcke, 
2009). 

However, little research has focused on how these factors influence discussion forum in a Chinese course. 
The purpose of this study is to investigate the effects of two instructional interventions: (a) different types of 
discussion activities, (b) different discussion leaders on asynchronous discussions in a Chinese online course. The 
results indicate considerable implications for designing effective online discussion activities. 

Method 



Participants 

291 in-service students enrolled in the online course, “Digital Photography and Image Processing”, in Fall, 
2009 at a metropolitan university in Shanghai, China. The semester had five months from Nov. 2009 to Feb. 2010. 
Students were all K-12 teachers. During the semester, they attended all the online discussion activities voluntarily. 

Discussion Groups 

First, based on previous research (e.g., Tenenbaum, Naidu, Jegede, & Austin, 2001; Kanuka et al., 2006), 
this study designed four types of discussion activities: sharing information, case study, debating, and problem- 
solving. In the first phase, students were randomly put into four groups and were asked to have discussions about a 
same topic of image processing in one month. In the second phase, all the students were randomly put into four 
groups too. Each group was assigned a same discussion topic about digital photography to discuss in one month. 
Four groups were led by different leaders, the instructor, a tutor, a peer, and no leader. The tutor had assisted the 
instructor to teach this online course for 3 years. The peer was chosen from the class. At the beginning of the 
course, the instructor posted the qualification to be a peer-leader in discussion board. Students who were interested 
in it could post a message to introduce and recommend themselves. The best one who mastered the discussion topic 



65 




and was good at communication was chosen and assigned the role of peer-leader to manage a discussion group. The 
discussion topic was posted by the instructor for all the four groups. The responsibilities of the discussion leaders 
were to post an initial message and answer questions. 

Instillments 

In the first phase, each posting was coded by using Interaction Analysis Model (IAM)’s five phases by two 
researchers (e.g., Wever et ah, 2009; McLoughlin & Mynard, 2009). LAM is a content analysis model based on a 
constructivist paradigm designed to detect evidence of knowledge construction. It was developed to understand and 
describe the processes of negotiating meaning and knowledge co-construction in a collaborative online discussion 
environment (Gunawardena et ah, 1997). This model has five phases: (a) sharing/comparing of information (PHI), 
(b) discovery/exploration of dissonance/ inconsistency amongst participants (PHII), (c) negotiation of 
meaning/knowledge co-construction (PHIII), (d) testing/modification (PHIV), (e) phrasing of agreement and 
applications of newly constructed meaning (PHV). One posting may achieve several phases of IAM. The highest 
phase that one posting achieved was used to identify its level. Two coders coded all the postings and decided each 
posting’s level together. Similar to the first phase, in the second phase, all the postings were also coded by IAM by 
both researchers to evaluate their levels. 



Results 

In the first phase, 36 students participated in four groups and they posted 72 messages. The postings were 
coded with IAM. A one-way ANOVA test revealed that level of knowledge construction were significantly different 
in terms of different types of online discussion activities (F(3, 68) = 3.42, p = .029) (see Table 1). A Tukey post-hoc 
test showed that debating discussion activity (M= 4.53, SD = 1.33) led to a higher level of IAM phases than sharing 
information discussion activity (M= 2.00, SD = 2.00). However, there was no significant difference between other 
types of discussion activities. 

Table 1. Mean scores and results of ANOVA test 



Group 1 : 
Problem 
Solving 
(N= 22) 


Group2: 

Sharing 

Informa 

tion 

(N= 8) 


Group 3 : 
Case 
Study 
(N= 8) 


Group4: 
Debating 
(N= 34) 


P 

Value 


Post-Hoc (Tukey) 


P 

Value 












Group 1 = Group 2 


.100 


M = 


M = 


M = 


M = 4.53 




Group 1 = Group 3 


.772 


4.09 


2.00 


3.25 


SD = 


.029* 


Group 1 = Group 4 


.873 


SD = 


SD = 


SD = 


1.33 


Group 2 = Group 3 


.644 


1.64 


2.00 


1.26 




Group 2 < Group 4 
Group 3 = Group 4 


.023* 

.429 



* p < .05 



In the second phase, 35 students attended discussions, and there were total 70 coded postings. One-way 
ANOVA indicated that level of knowledge construction were not significantly different in terms of discussion 
groups led by different leaders (F(3, 66) = .329 , p = .804) (see Table 2). 

Table 2. Mean scores and results of ANOVA test 



Group 1: 
Instructor- 
lead 
(N= 8) 


Group 2: 
Tutor-lead 


Group 3 : 
Peer-lead 


Group 4: 
No leaders 


P Value 


(N= 18) 


(N= 22) 


(N= 22) 




M = 4.25 


M = 3.56 


M = 4.00 


M = 3.45 


0.804 


SD = 1.50 


SD — 1.74 


SD= 1.73 


SD= 1.81 
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Discussion and Conclusion 



The limitation of this study is the sample size was small. Since the discussion activities were not required, 
less than half students attended the discussion. Chinese students are used to teacher-centered and test-driven 
instructional format (Wang & Kreysa, 2006). On the other hand, influenced by Confucian Heritage Culture, 

Chinese students are cultivated to revere authority, maintain harmony, and avoid conflicts in public (Chiu, 2009). 
When they are asked to present perspectives on an issue or discuss with each other, they always feel nervous and 
reject to attend them. How to motivate more Chinese students to join online discussions is really a big challenge for 
Chinese online teachers. For this research, more data will be collected in the future to support and validate current 
findings. Rewarding students extra points, increasing the weight of the score, or extending the discussion time will 
be the considerable strategies to encourage more meaningful participation. 

With the limitation, we found that debating discussion activity led to higher level of knowledge 
construction than sharing information discussion activity. Although Chinese students seek harmonious learning 
environment and tiy to avoid conflicts in public, more students were willing to attend debating discussion, rather 
than only sharing information in this study. This result was consistent with previous research. Kanuka et al. (2006) 
indicated that students who engaged in the WebQuest and debate activities showed higher participation and 
cognitive presence than other groups. Debate is viewed as an effective activity to facilitate students achieving 
higher-order thinking because it could provide students with more opportunities to find supportive evidence and 
synthesize others’ opinions. The finding indicates that the type of discussion activities is a considerable factor when 
designing an online discussion. 

The second finding showed that there were no significant differences among the four groups led by 
different discussion leaders. Although Chinese students are used to teacher-oriented instructional format, the 
instructor-led group did not have higher level of knowledge construction comparing other groups. Based on these 
findings, to encourage more meaningful participation and achieve higher level of knowledge construction on 
discussions, designing an appropriate discussion activity maybe more important to be considered by Chinese online 
teachers. 
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Abstract: With U.S. doctoral programs having attrition rates averaging 50% according to some 
studies, increasing attention is being given to improving retention. This case study analysis 
examines how one educational technology doctoral program has used design, technology, and 
formative evaluation toward a goal of full retention for all students admitted. Narrative data from 
three students facing circumstantial events that would normally have required them to leave the 
program illustrate the program’s success to date. 



Introduction 

While much research in distance learning focuses directly on teaching and learning, increasingly designers 
are recognizing that successful efforts require development of learning environments (Huang, 2002; Zhao, Lei, Yan, 
Lai, & Tan, 2005) and the critical role of affective context (Craig, Graesser, Sullins, & Gholson, 2004; Isen & 

Reeve, 2005; Pierre & Oughton, 2007). Establishing such classroom settings that improve student attitude and 
motivation are important to enhance student retention efforts. 

The purpose of this case study is to understand how technology can provide support for successful student 
persistence in a doctoral program. The study examines an educational technology Ph.D. program at the University of 
Hawaii at Manoa and focuses on the experiences of three students who were facing life events that would normally 
have required them to leave the program. Through narratives of their experiences and commentary by other students 
in the program, issues of technology affordances are described and evaluated in terms of success in supporting 
student retention. In particular, the study shows how distance students can be integrated into face-to-face classes. 

Background 

To date comprehensive statistics are not kept for doctoral program retention in the U.S, but the available 
data indicate a significant proportion of students do not finish. A study by the Council for Graduate Studies (2008) 
showed that the average student attrition rate was 23.6% by the fourth year after admission in over 400 doctoral 
programs from 29 universities both public and private. Earlier studies indicated higher rates looking at a broader 
range of fields with rates over time showing only about half of students entering doctoral studies as completing 
(Golde & Dore, 2001). While no reports have examined attrition in education doctorates beyond studies of single 
institutions, there is little to suggest the field is different. 

Research results from studies on doctoral student attrition have indicated multiple and complex reasons for 
the high rates of students who do not complete through dissertation, including situational, program-specific, 
cognitive, and affective or personality factors (Golde, 2005; Green, 1997). While common wisdom holds that 
students who leave may not have what it takes, accrued research evidence shows that there are few differences in 
characteristics such as GPA, test scores, or other personal factors at the time of admission between those who finish 
and those who dropout, suggesting that program and institutional impacts play an important role (Council of 
Graduate Schools, 2009; Lovitts, 1996; Maher, Ford, & Thompson, 2004) 

Multiple solutions have been proposed to the attrition problem, including reports from an ongoing study by 
the Ph.D. Attrition and Retention Project (Council of Graduate Schools, 2010). Based on exit interviews by program 
completers, factors reported as critical to success are less an issue of academics and more about environmental 
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factors, including not only the institutional roles of financial aid and advising, but also non-financial support by 
family (Council of Graduate Schools, 2009). In a recent study, socialization and integration were noted as major 
components to establish both student support and expectations, establishing key aspects of the affective environment 
(Gardner, 2010) and echoing the findings in earlier studies on attitudes and the impacts of institutional and 
disciplinary settings (Cooke & Sims, 1995; Lovitts, 1996). Berge and Huang (2004) note that in the various models 
of academic persistence in the literature, variables accounting for attrition have included personal, institutional and 
circumstantial factors. They proposed a framework that includes the following: 

• “Encourage commitment (personal goal commitment, institutional initial and ongoing 
commitment) 

• Enhance integration (management and support services that enhance academic and social 
experiences) 

• Improve delivery systems (delivery of instruction and support in online, blended and in-person 
settings, e.g., instructional support services, student support services, staff development on 
proactive academic advising; institutional network) 

• Increase person-environmental fit (ease stages of transition, facilitate person-institutional, person- 
circumstantial and institutional-circumstantial fit) 

• Improve outcomes (academic outcomes such as academic performance and intellectual 
development, psychological outcomes such as perceived utility and satisfaction)” (Berge & 

Huang, 2004) 

Because issues of student persistence in doctoral studies are being examined both externally by policy 
makers and internally by universities, doctoral programs are being urged to examine their design and improve 
factors that will lead to retention and completion. This concern was noted and addressed when the University of 
Hawaii at Manoa Department of Educational Technology (ETEC) started its new doctoral program with a goal of 
graduating all admitted students. 

In 2007, ETEC began its program as a specialization area within the existing education Ph.D. in the 
College of Education. We are now in our fourth year, having admitted 32 students from diverse backgrounds and 
geographic locations. Students are a mix of full and part-time, with core classes being held primarily face-to-face in 
the evenings and most electives online. The advantage to being a relatively new program is the ability to benefit 
from the experiences of other programs and the research that has been done on doctoral program success over many 
years. The ETEC doctoral program is still young and lacks program finishers. However, after four years the program 
has not lost a single student and continues in its goal, perhaps unrealistic but meaningful, of 100 percent reaching 
degree status, far superior to figures in the national studies cited above. 

While still early to determine our success at our admittedly lofty goals, this paper reports on strategies 
related to student retention using technology as a support for both community and learning. The case features the use 
of new classroom technologies through the experiences of three doctoral students, the co-authors of the paper, who 
provide voice for student concerns that arise when they can no longer be on campus for a location-based program. 

Methods 

The purpose of this qualitative case study research was to determine how the application of program design 
and technology impacted student retention in the ETEC doctoral program. The research used a narrative approach to 
gather data from three students who faced situational crises that forced them to be off-campus and would normally 
have required each to take a leave or withdraw from what is a campus-based doctoral program. 

Data collection involved responses to open-ended questions on program strengths and specific supports that 
assisted in continuing their studies from a distance. Each of the students wrote a narrative describing their situations 
and interfaces with the doctoral program during off-campus residency. In each story, this involved living on the 
mainland due to personal and family crises, which created both distance and time issues. In addition, twelve students 
from the face-to-face contingent provided additional data by briefly responding to emailed questions about their 
classroom experiences from a campus-based student perspective. 

Data analysis involved an inductive approach to coding and thematic grouping of the resulting responses 
with a baseline on categories derived from frameworks in the retention and attrition studies. Results are reported 
primarily in narrative form to provide voice for the student perspective. 
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Results 



Three students are the focus of this study, each of whom moved to the mainland after beginning the 
campus-based Ph.D. program in educational technology at the University of Hawaii at Manoa. Two of the students 
were entering their second year, while the third was off-campus after only one semester. In all three cases, the 
variables impacting decisions to leave campus involved what Berge and Huang (2004) classify as personal 
circumstantial variables external to the institution. These are the life, work, and family/socio-economic 
circumstances which are beyond the control of either the student or the program, but are particularly prevalent 
among the adult students in graduate programs. One student’s situation gives a sense of the complex and 
individualized nature of the problem. 

I took leave from my job and came to the University of Hawaii (UH) in August of 2008 to start my doctoral 
program in educational technology’. I was excited to be starting work on a prestigious degree, to be 
studying in depth a subject that I enjoy, to be away from work, and to be in Hawaii. My plan was to be in 
residence at UH for four semesters and then return to my job. The first semester went well. I met and 
started working with my cohort for the first time; they are an excellent group of students. We all took 
classes together from our mentor. Our entire cohort learned from her how to be academics and what was 
to be expected from us as our programs un folded and we marched towards our dissertation. It was a good 
beginning. 

The second semester started uneventfully with our cohort reuniting after the holidays and picking 
up where we had left off. It was soon after that I learned a vision problem I was experiencing was caused 
by a degenerative disease and hard times hit me. Without specifying the details of my health problems I can 
honestly say that I went through the worst time of my life that semester. I didn ’t know if I could make it 
through school or if I would have to withdraw and head home to New York for recovery. It was the support 
I received from my fellow students and the faculty that allowed me to stay. In particular, one member of my 
cohort and an excellent friend, nursed me through some of the worst times. And our faculty’ mentor 
convinced me that the ETEC Doctoral Program at UH would support me and “we want you to finish. ” The 
belief in me by these two people, and the whole ETEC ‘Ohana’ (extended family) allowed me to stick it out 
through the semester. In May of 2009 1 left for New York to take some time with family to help me adjust to 
my changing health. 

In most campus-based programs, students such as these three would have been required to take a formal 
leave of absence because of inability to continue with required coursework and distance from faculty advisors. This 
is the case in all other disciplinary doctoral programs in the UH College of Education as none are distance programs 
and most have no online doctoral course offerings. In fact, none of the required research courses are currently 
available online and these are a major part of doctoral coursework in the first two years. In the event that 
circumstances forced a student into a prolonged residence off island, the student would be forced to leave the 
program if they had not yet completed required courses. 

However, with a goal of student retention, students in the ETEC program were encouraged to continue 
without interruption with assurances that solutions would be invented on the basis that department faculty and 
students were experts in distance learning and dealing creatively with change. As one student wrote, 

Leaving Hawaii did not mean that I had to leave my doctoral program. On the contrary, I was encouraged 
to stay and carry’ on my studies from a distance. Even though the program was not officially online, I was 
assured that I would be able to complete my studies online. The coursework fell into line. - Distance 
student 

The students identified with many of the program aspects that were intended to ensure retention, including 
committed relationships with faculty, particularly their advisor, and support from their peer cohort of students. This 
strong sense of community and the multiple ways it developed initially through a combination of face-to-face and 
technology-enabled interaction established a baseline that made it possible to remain connected when a student was 
no longer on campus. Throughout the program, community building was supported by technology for 
communications and collaboration. Such tool use went beyond classroom activities, connecting the ETEC 
community both socially and intellectually. As one student noted who joined the program in its second year, 2008, 
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I was fortunate to make a strong bond with both faculty and my cohort during the year that I was able to 
reside in Hawaii. The Ning [a social networking] site that our advisor set up prior to the beginning of my 
course work enabled me to make connections prior to meeting people F2F. I felt as if I knew everyone prior 
to actually meeting them in person. Great idea to build an online community’ as a form of introduction. The 
first cohort welcomed our cohort and were assigned to be our mentors. I met the members of the first 
cohort during an informal book club setting. It was relaxed and welcoming to be invited to our advisor’s 
home for the event. I am not sure I would have been able to continue my education from a distance without 
the support and friendship of both faculty’ and my peers. Iam not sure a fully online program would create 
the support system that I have now. I was able to spend time both socially and professionally with this 
support system. It enabled me to build trust and a feeling of belonging. 

Innovative application of distance technology played a key role in the ability to continue coursework and 
research, building on the same technologies used as the core of the program for teaching, learning and social 
networking. For example: 

My ties with Hawaii continued from New York. I kept up with the members of my cohort, some of whom 
were also in my Disability Studies class, through email, chat, and Facebook. I came to know other UH 
students through my TA-ing the undergraduate course. That experience also led to strong bonds with the 
members of the teaching team with whom I met weekly online and began a research project. I also kept up 
with the happenings of the ETEC Doc Students though a social network site, Ning, specifically created to 
foster bonds among the students. I was able to participate in Book Club, a monthly event sponsored by our 
mentor that is held on campus, through a discussion forum in the Ning. I was still a part of the ETEC 
Ohana [extended family] . - Distance student 

The greatest challenge involved adapting courses that served campus-based students to include these new 
distance learners. Both F2F and distance classes at UH use asynchronous tools for collaboration, resource sharing, 
and class administrative activities. UH has a widely deployed course management system, known at Laulima, which 
is a customized version of Sakai, that is used in both distance and campus classes. In addition, Google Apps for 
Educators provides collaboration tools and shared productivity applications, while the institution provides the most 
common communication channels of email and listservs. Students choose to supplement these with free Web apps 
such as Skype, Facebook, Twitter, and SecondLife. 

While interactive video-system based courses have long included multiple locations, classes specifically 
established for the bricks-and-mortar classroom do not have similar models for bringing in only a few students from 
any location using ubiquitous technologies. However, as noted in one study, “The problem involving retention of 
students is not due to an isolated factor that can be ‘fixed,’ but rather imagination and care must be used to carefully 
select interventions that are needed at various points” (Berge & Huang, 2004). 

In the case of ETEC doctoral classes, the solution was the use of a synchronous audio conferencing and 
course tool already being used at the UH College of Education for distance classes, a commercial product called 
Elluminate*. Interestingly, because this tool is advertised for distance learning, using it within the F2F class required 
rethinking the affordances. While initially Elluminate was used with a single computer projected for classroom 
sharing, it became more useful when it was used by each student individually on their own computers regardless of 
location. 



I like that [ the instructor] has everyone log in to Elluminate even though they are F2F. Last semester I had 
a class where I was the only one participating from a distance. [One student] logged in to Elluminate to 
keep me company. Often I lost audio and if [he] wasn ’t logged on with me I would have been lost in 
cyberspace. It is difficult to make accommodations for one lone distance learner. I appreciate the effort and 
compassion from everyone who has tried to make it work for me. I feel so fortunate to be able to continue 
from the mainland. - Distance student 

Students were required to have laptops for class, but to best create a shared environment, each student 
logged into Elluminate on their own laptop. As a result, both F2F and distance students had access to all the tools in 
Elluminate during each session. One accommodation was using a single USB conferencing microphone/speaker for 



* More information on Elluminate can be found at the company’s web site, http://www.elluminate.com including the use of a free 
virtual room for up to three people. 
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all audio in the classroom. When a distance student was speaking, their voice was heard on the classroom speaker 
and students muted their own laptops to prevent audio feedback. Students in the classroom were both positive about 
the ability to keep their classmates within the program but recognized the importance of more synchronous 
inclusion. 

I think that Elluminate was a great way for the distance students to stay connected in class by participating 
in class discussions and group work. The greatest strength of Elluminate was its ability to foster 
communication between distance students and the class. The ability to have break out rooms was also a 
feature that was utilized often. Some of the drawbacks, however, is that things did not always go smoothly. 
Trying to work in groups was at time difficult because we had to keep the mic open so that the distance 
student could hear the discussion. Due to the limited range of the mic, there were times where the distance 
students couldn 't hear what was being said and things would have to be repeated either in voice or in the 
text chat. Having to repeat yourself also happened often in whole class discussions too. Also, having 
multiple distance students in the class would often require the rest of the class to split up into different 
rooms so that multiple open mikes could be used. - F2F student 

I think that it works very well, especially considering the great distance between us. One strength is that all 
our voices can be heard in real time. Ideas can be developed and shared via text or vocally. This dialogue 
is very similar to any shared between people sitting in the same room, or F2F. That is a distinct advantage. 
A drawback may be that there is no visual, but since some of the mainlanders may be in the pajamas that 
might be good. But we would all like to see Germany or Taiwan, where others have been :) - F2F student 

Once Elluminate was deployed within the F2F class, it not only allowed synchronous participation by the 
three mainland-based students but enhanced the range of tools for all, adding some capabilities which have required 
expensive hardware and software to implement but are seen as desirable in campus classrooms. Some useful feature 
of Elluminate include: 

• Audio and video recording of class sessions - many campuses have spent large sums to equip 
lecture halls to do this but Elluminate had all the needed tools. This proved to be one of the 
greatest advantages, as both campus and distance students appreciated the ability to go back and 
review class presentations which are rarely captured in most college classrooms. 

• Polling - a tool within Elluminate had similar capabilities to remote clickers for rapid feedback 
and quizzing. 

• Back channel discussions - students could chat about the lesson and interact throughout with 
anyone in the class. This was also used to instantly contribute web links and journal article titles 
that related to a topic under discussion, increasing sharing of resources among the students. 

• Screen sharing - anyone could immediately share a web site or document from their own 
computer without added hardware or software 

• Whiteboard - by using the interactive whiteboard provided in Elluminate, PowerPoint 
presentations, images, and simple drawing tools could be used at any time by all class participants 
and recorded with the class session. 

• Video - while the tool uses a computer’s web camera for video, this was used only sometimes as 
bandwidth issues caused problems. Seeing a talking head was not always necessary but it added 
personality to see who was talking. Flowever, it was a small image and not equivalent to sitting in 
a classroom viewing the whole space. 

• Guest lectures - experts could easily participate and were brought into the classroom from 
multiple locations, both locally and internationally, with no specialized video conferencing 
equipment. 

When asked what features were helpful, students had many positive comments: 

Synchronous audio and chat providing multi-sensory access to both faculty and student colleagues. 
Synchronous whiteboard providing real-time slide presentations. - Distance student 

The chat box is an especially useful tool as it allows simultaneous conversations with multiple peer 
students allowing for ways to socialize and keep in touch with classmates. It is also a very’ useful tool to ask 
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for quick clarification from a classmate when the audio transmission may have been unclear, for example, 
about a class assignment, or group assignments. In addition, it is strongly suggested by this student that it 
is made a standard practice that an "in-class partner" student be assigned to monitor the chat box for 
questions from the student at a distant, and also periodically ask how the distant student is doing or if there 
are any questions to make sure they are still connected and to bring them into the classroom. -Distance 
student 

The tools were not only useful for distance students, but as these were implemented for support of these 
off-campus colleagues, the F2F class began to experiment with new uses. This included both in-class uses and as 
support for class tasks outside of the classroom. 

My favorite thing about Elluminate is that its the best online collaboration tool that I've used. I once 
collaborated on a group project via Skype, but its application was too buggy and would often crash. We 
ended up switching to Elluminate because Skype could not support my group's work. I also like being able 
to have public text chats and private text chats. Multiple mics is also a feature that I use often in group 
work. For presentations, we often utilize the white board and its tools, as well as the polling feature and 
timers to increase interactions between the speakers and group. -F2F student 

I like it when the class leader allows us to divide into small groups and discuss and then come back to the 
large group. I like that Elluminate allows for the loading of a content based power point, or just the simple 
sharing of ideas like a chalk board in a class room. - F2F student 

Further, with all sessions done in Elluminate, F2F students who needed to miss a class because of illness or 
other issues were also able to join the class virtually. This proved to be beneficial in general to classes and did not 
lead to lowered attendance, as other issues drew the campus students. In fact, one aspect all students noted was that 
the non-class related camaraderie, food (students typically bring food to share since the course meets over the dinner 
hour), and show-and-tell of the latest technologies was missed, even by campus students who had to be absent for a 
class. 



Due to my current job, I travel quite a bit during school year and I had to take advantage of Elluminate to 
attend a F2F class. The most recent experience was a seminar session where I participated from South 
Korea. Initially, I had to figure out the time difference so I can participate at the right time but once that 
was figured out, it was as if I was there. At times, it was difficult to hear what the classmates who were 
present F2F were saying but for the most part, it was a very’ engaging experience. -F2F student 

I haven't taken any advantage of Elluminate to attend a F2F class. I just missed one online lesson and the 
recording helped me catch up with my classmates. - F2F student 

Some aspects of positive socialization of cohort model lost in distance learning model (e.g. distant learner 
not able to participate in class celebrations, sharing of food, etc.). ...In F2F class, socialization occurs 
before, during, after, and outside of class. Deepening bonds of friendship and trust create a safety’ net for 
the researcher. It becomes easier to ask "dumb" questions of this tight knit group. The group acts as a 
cheer squad for you as you near deadlines. There is no one to commiserate with at the beginning and 
ending of each class. ([One student] and I groan and moan about "the readings are killing us!") -Distance 
student 

Not all aspects of the distance arrangement in the F2F class worked perfectly. Even those within the 
classroom noted that it was too easy during intensive interactions to remember that there were students not in the 
room. For example, the chat feature had both pros and cons, as described by one of the F2F students, 

It worked well for those in the classroom in that it allowed for students even in F2F to 'chat' and 
collaborate with each other without making any spoken noise that would disrupt the instructor and those 
whose attention are fully on other things. One of the strengths is that it allows for any student, whether in 
F2F or at a distance, to be part of and included in the 'chat' conversations since the chat feature levels the 
playing field for all students. However, one weakness would be that the students at a distance would have 
to make an effort to continue manipulating the features in Elluminate, ex. happy face, chatting, clapping, or 
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anything deemed appropriate to keep those in the F2F classroom from indirectly 'leaving them out' of 
conversations going on. - F2F student 

Students recognize that the applications to support distance learners remain underdeveloped, with both 
hardware and software issues, with particular issues around audio quality. Adequately picking up all talking in a 
classroom without issues of feedback and sound delay remains problematic. 

Multiple microphone and speaker systems need to be investigated and improved. Distance listening is 
extremely fatiguing in a group conversation setting. The spontaneity > of discussion means that multiple 
people speak at once in the f2f location. Facial cues and gestures support these communications for the 
F2F group, however, these communications are often garbled for distance students. - Distance student 

On several occasions I have attended a F2F class via Elluminate and overall it has been a good 
experience. As long as the microphone in the class is picking up the conversation/dialogue it is a good 
experience. On occasion, it can be a bit of a challenge when multiple people talk at once, over each other, 
or not close enough to the microphone. This can result in "garbled” sound that is hard to understand. - 
F2F student 

One problem that arises is with time differences, particularly between Hawaii and the mainland. In fact, one 
student now joining from Germany in Fall 2010 faces a 12-hour difference, so that the student has early breakfast 
while the Hawaii students are having dinner during the evening class. This requires commitment when students have 
to accommodate within schedules of family and work, and class times are commonly inconvenient. As one student 
indicated: 

Having class until 11:30 pm is brutal. Knowing this is my last semester of full time course work keeps me 
going. I could not sustain this schedule throughout the entire program. - Distance student 

The contrast to the strong interactions established in the face-to-face setting can be missed, even when a 
student indicates a strong desire to persist and technology enables continued communication with others. The ebb 
and flow of face-to-face conversations is more difficult when technology stands between the communicators. This 
appears as an unresolved concern directly within classroom activities as well as beyond. 

When the class did group work and lam a member of a group from a distance it is difficult to "insert" 
myself into the conversation because I am not always sure whether I know what the conversation is. 
Sometimes, even in a small group of four classmates, there might be two conversations going on. When 
there is a time limit, the group is intent on getting the work done with the people in the room and not 
always watching the chat box for suggestions I am making. - Distance student 

Students felt supported both academically and affectively through their personal challenges. Each was 
positive about the individual support received but also had many suggestions for the future to improve 
communication. They noted that the close feeling of being face-to-face was missed but academically they could 
continue. As one student stated, 

I think that a majority of the reason that I was successful in continuing with the ETEC Doctoral Program 
online was that I had a year in residence at UH. My first year introduced me to the people and the culture, 
both academic and social, which comprised the program. I felt connected to UH because I could easily 
envision the faces of the people and the places on the island. It was the connection that supported me over 
the distance. - Distance student 

This was also noted by another: 

lam missing the daily conversations and the ability to communicate F2F. The time change makes it 
difficult to stay in sync with the rest of the group. Planning a time to meet for group work is a compromise 
for everyone. I am either way ahead of everyone or I feel as if I am behind. I still communicate using social 
networking sites, however, the communication is less personal and less frequent. Iam starting to feel left 
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out, even though numerous people have made a point of trying to make me feel included. It is like a long 
distance romance. - Distance student 



Conclusions 

When a decision is being made to persist or dropout, “it is made by the individual student, influenced by his 
or her personal circumstances. It is based upon the student’s continual cost/benefit analysis of all social, 
organizational, economical, and psychological factors like those resulting from perceived opportunity, relevancy, 
stress, responsibility and satisfaction within the educational context’’ (Berge & Huang, 2004). These are responses 
with strong roots in the affective and environmental aspects of the learning environment. When the program has 
built both strong ties among faculty and students, and provided a system of supports involving advising and 
communication, those factors that can be controlled when circumstantial factors impact student lives allow 
resolution that will lead to persistence. In the examples given, both program design and creative intervention 
established the framework for support through difficult life circumstances, leading to persistence by these students. 

In particular, new applications of technologies already used in the program allowed students to continue with 
coursework and feel a part of their doctoral cohort even when they were no longer on campus. Technology 
supported both direct academic progress and community built earlier during the campus experience. This success 
holds potential for establishing a hybrid doctoral program designed to support non-campus based students, a great 
need in Hawaii where the only major public institution with graduate programs serving the state is located on a 
single island in the state chain. 

The results of this small study are now being evaluated to incorporate the students’ feedback into program 
design. While one student has now returned to campus, two remain off island with circumstances that will preclude 
more than short visits during their remaining time in the program, yet all are making positive progress that matches 
their on-campus peers and are committed to persistence. Since this study was done, several additional students have 
left before completion of the program, but in these cases course work was completed allowing more independence in 
pursuing research planning and implementation towards a dissertation. While this pattern of departure from campus 
after coursework is not uncommon in education doctoral programs, it is one of the factors identified as positively 
impacting attrition and therefore an area for continued program adjustment. To meet these new issues, technology is 
being used to ensure both social and advising connections are maintained, including virtual participation in a 
dissertation support group that meets with a faculty mentor. The longer term issue of program completion remains 
unknown but to date all are continuing and actively working on their research. 

While this research began as one to study technology for supporting distance students, the results also are 
meaningful in terms of broader issues of retention of doctoral students. Although faculty have often focused on class 
work and research in the development of doctoral students, the design of program environments and conscious 
understanding of affective supports may have even greater relevance for ensuring student persistence. The 
implementation of intentional affective strategies have often been ignored in graduate education because of attitudes 
that equate adult learning solely with cognitive functioning (Craig, et al., 2004; Pierre & Oughton, 2007). Even at 
the most advanced levels of education this element can be seen as one that will have major positive implications 
when intentionally addressed. 
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Abstract 

The purpose of this study was to investigate how students perceive the Quality Matters ™ (QM) standards relative 
to their success in online classes. Quality Matters (MarylandOnline) offers a rubric with 45 standards they believe 
are important, very important, or essential to an efficiently and effectively designed and organized course. The 
rubric that Quality Matters uses is derived from an extensive review of the literature, but there is little research about 
how students rank these standards relative to their perception of success in an online course, and how age, online 
course experience, and major are related to their perceptions of the QM standards. A convenience sample of 77 
participants from two mid-sized universities found that they perceived 26% of the Quality Matters elements as 
essential, while a Spearman’s Rho showed no significant correlation between age and student perception of the 
Quality Matters standards, and between the number of online courses previously taken and the QM standards. An 
analysis of variance was performed between student major and their perception of the Quality Matters standards 
showing no significance, although a small, practical effect size was observed. Results from this survey can be used 
to inform online instructional design and course organization practices. 

Keywords: online course standards, online course design, quality matters, student perception of online course 
designs 



The research literature is replete with studies examining student satisfaction in online learning 
environments relative to their interactions with other students, the instructor, and the course content (Anderson, 

2003; Moore, 1989; Swan, 2001; Thurmond, Wambach, & Connors, 2002). Additionally, there are many studies that 
present evidence of the qualities found in effective online courses. (Burnett, Bonnici, Miksa, & Kim, 2007; 
Drummond, 2008; Hemy & Meadows, 2008). However, there appears to be less research regarding how students 
perceive their success in an online course relative to the course’s instructional design and course organization. 

Course instructional design and organization refers to the way the online course is structured, where links 
and areas are found relative to content access, discussion board participation, online chat rooms, and where students 
may locate additional recourses (within or outside of the online class). As Simonson, Smaldino, Albright and Zvacek 
(2009) offer, students benefit when they have a clear understanding of course organization and student 
responsibilities. The authors further state that it is essential for students to “understand how the course will function 
so that they can be better prepared to participate” (p. 190). Aligning with course design is the role of instructional 
design in order to make learning more efficient, effective, and less difficult with a focus on the learner rather than on 
what content to cover (Morrison, Ross, & Kemp, 2007). Efficient and effective course organization in an online 
class means providing clear directions for navigation, and pronounced directions and cues that support student 
movement from one place to another within the course. The use of a site map, descriptive labels, and clearly marked 
work areas that inform the students where they are in the online course support efficient navigation. Course content 
and tools grouped appropriately under clear, consistent headers along with a concise table of contents also support 
efficient organization. For example, presenting students with a course structure page (like a site map) to explain how 
the course is organized will help them quickly see where key content and participation areas are found. Effective 
course design also means providing students with an explanation about the tools they will be using during the course 
and how to use those tools in the course (Simonson et al., 2009). While there are many ways to evaluate the quality 
of the design of an online course, one way is through an established rubric. 

Quality Matters ™ (QM), from MarylandOnline, offers online instructors such a rubric which measures 
critical components of a well designed online course. A rubric can be an effective evaluation and standardization 
tool that “combines evaluative criteria with scales that explicitly define standards of performance” (Kuhs, Johnson, 
Agruso, & Monrad, 2001, p. 55). The QM rubric clearly outlines criteria required for an efficient, effective and 
engaging online course. For example, under the Quality Matters category of Course Overview and Introduction, one 
of the rubric items states, “Instructions make clear how to get started and where to find various course components” 
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(MarylandOnline, 2008, p. 2). This standard is further elaborated with annotations providing explicit details about 
this item in addition to an example of what it looks like when implemented. The QM rubric as developed, fits the 
definition of rubrics provided by Cooper and Gargan (2009) who describe rubrics as standards with descriptive 
statements illustrating how the standard is to be achieved. Jonsson and Svingby (2007) found that rubrics have the 
potential to improve instruction because expectations and criteria are fully developed which also facilitates feedback 
and self-assessment. 

Because the QM rubric presents standards upon which online instructors are to focus, it offers a consistent 
way to evaluate a course’s design and organization components. It ensures that online classes participating in the 
program or following the QM recommended standards will be judged using the same criteria; ensuring consistency 
and quality in online course design for colleges, universities, and organizations. 

The Quality Matters (QM) standards are based on best practices and instructional design research which 
started from a Fund for the Improvement of Postsecondary Education (FIPSE) grant in the fall of 2003. The grant 
work resulted in the creation and implementation of a process to certify the quality of online courses and online 
components. The grant expired in 2006, but the organization decided to continue with their work towards “inter- 
institutional quality assurance and course improvements in online learning” (MarylandOnline, n.d.) As a result, the 
Quality Matters program currently has over 400 subscribers across 30 states ranging from four-year colleges and 
universities, technical colleges, community colleges, to other academic organizations. These institutions and 
organizations use the QM standards to evaluate the quality of the design and organization of their online and hybrid 
courses (MarylandOnline, 2008). These standards help guide faculty, instructors, and course developers to build 
efficient, effective, and pedagogically sound online courses. Faculty and instructional designers are trained and 
certified by the QM organization and its designated master trainers to review and evaluate online courses in 
accordance with these standards. 

The focus of QM is to support student learning by offering standards through a faculty-driven, peer- 
reviewed process; a process that allows courses to be formally or informally reviewed using the QM standards. 
Instructors can use the rubric to guide the development of face to face classes and online classes, while peer 
reviewers use the rubric to examine the online course (or classroom course design) and to provide detailed feedback 
to the course developer about whether or not the course met the quality standards. Each of the 40 standards has a 
point value (assigned by the Quality Matters organization) depending on its relative importance to 
course/instructional design. Seventeen of the standards are considered an essential part of an online course and have 
a point value of three. Eleven standards are considered very important with a point value of two, and 12 are 
considered important with a point value of one. The maximum number of points that a course can earn is 85 and for 
a course to meet QM standards it must have an overall score of 72 points (MarylineOnline, n.d.). 

These 40 elements concentrate on course design and not on course delivery or the academic content of the 
course, and are grouped into eight categories as follows: 

1) Course Overview and Introduction - Addresses how the overall design of the course is made clear to 
students at the beginning of the course. 

2) Learning Objectives (competencies) - Ensures that the course objectives are clearly explained and easy to 
understand and follow. 

3) Assessment and Measurement - Ensures that assessment strategies used to evaluate student progress are 
supported by the stated learning objectives and are an integral part of the learning process. 

4) Resources and Materials - Addresses whether the instructional materials are comprehensive enough to 
achieve the course goals and objectives, and that the materials are prepared by qualified persons. 

5) Learner Engagement - Ensures that there is meaningful interaction between the instructor and students, 
among the students and between the students and course content. This standard is designed to support motivation 
and foster intellectual commitment to the course. 

6) Course Technology - Addresses navigation in the course as it supports student access to the materials and 
that technology is also used to enhance engagement with the materials. 

7) Learner Support - Ensures the course provides access to institutional resources designed to support student 
success. 

8) Accessibility - Ensures students have access to all of the course components. 

The 40 standards presented in the form of a rubric provide course developers, instructors, faculty, and peer 
reviewers with an efficient and clearly described criteria list for developing and evaluating the design of online 
courses. 

Although the QM rubric is supported by an extensive literature review regarding best practices for online 
instructional design and course organization (MarylineOnline, n.d.), nowhere does Quality Matters address how 
online students perceive these standards relative to their success in the online learning environment. Therefore, the 
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purpose of this study was to examine student rankings of the QM rubric in terms of how these elements support their 
success in an online course. The study focuses on the following research questions: 

1. In what order do students rank the Quality Matters standards in terms of the items’ influence on their perception 
of success in the class? 

2. Is there a relationship between age and student perception of the Quality Matters items? 

3. To what extent does the number of online classes a student has previously taken explain their perception of the 
Quality Matters items? 

4. Is a student’s major related to their perception of the Quality Matters items? 

Rationale 

Creating online courses that support student learning are critical if institutions are to provide high quality 
online learning experiences, maintain accreditation, and justify their distance learning programs (Simonson, et al., 
2009). In order to do these three things well, institutions need to consider what the students themselves believe are 
key elements in course design relative to a successful online learning experience. It is possible to build sections or 
add course features in the online course only to find students do not avail themselves of those areas if they are hard 
to locate and not perceived as central to their success in the class. Creating areas that students do not use, or areas 
that do not support student learning, wastes instructor time and can add unnecessary elements to the online learning 
environment; thereby increasing complexity, cognitive load, and opportunity for confusion. Perhaps students who 
have more experience with online courses or more maturity may not find the QM standards as important to their 
success. Regardless, it is important to ask the online student how he or she perceives the QM standards in relation to 
how those elements support their success in online classes. 

Although the MarylandOnline organization believes they have provided comprehensive standards relative 
to designing effective online courses, they do not discuss what course design and organization elements students 
believe are critical. This study begins to bridge that gap by examining how students perceive the standards found in 
the QM rubric supportive of their success in online courses. The results of this study offer a student perspective of 
the QM standards with the potential to inform online instructors and course developers regarding the design of more 
efficient and effective online courses. 

Literature Review 

The Sloan Consortium (Sloan-C) recently reported that over 4.6 million students were taking at least one 
online course in the fall of 2008, representing a 17% increase over the number reported in 2007. Additionally, more 
than one in four college and university students now take at least one online course (Allen & Seaman, 2010). With 
numbers like these and continued growth for the online learning sector predicted, it becomes crucial that institutions 
pay attention to the quality of their online courses and that they implement some form of quality standards to ensure 
consistent excellence. 

A salient aspect of online learning is the interaction online students have with their online peers, the course 
instructor, and the course content (Moore, 1989; Swan, 2001; Thurmond & Wambach, 2004). But efficient online 
interaction cannot take place if the course design is faulty and students stumble in the course trying to find needed 
materials, resources, and course objectives. Poor course design can result in frustration, decreased student 
satisfaction, lost effort, and attrition (Yang & Cornelius, 2004 ). 

When students begin an online class, their immediate impression of the class is formed by the course 
design and layout, and the visual appearance of the course that greets them when they first log on. It is their first 
opportunity to see the virtual classroom and hence judge how they think the class will support them and their 
learning efforts. As such, instructors, faculty, and online course developers need to structure academic content in a 
“form suitable for study by distant learners” (Moore & Thompson, 1990, p. 4). Drummond (2008), in his report on 
student success in online learning, stated that a clear, well-structured and engaging presentation of concepts is a key 
component of a successful course (p. 44). Others discuss the need for clarity, consistency, and ease of use across the 
design and organization of online courses (Cheul, Junghoon, & Youngchan, 2006; Rhee, Moon, & Choe, 2006; 
Swan, 2001; Young & Norgard, 2006). Young and Norgard’s study asked students about their preference for 
consistent structure and navigation across all online courses and 92% of the students queried agreed that a common 
structure across all online courses would be helpful to them. 

Song, Singleton, Hill, and Koh (2004) found that student success in the online environment was predicated 
on course design and comfort with online technologies among other things. They reported that 82% of their study 
participants felt that the design of the course and comfort with online technologies influenced their success in the 
course (p. 65). Furthermore, the authors discovered that the design of the course was the most important factor 
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relating to students’ satisfaction with the course. Goals, clearly stated objectives, and course organization were also 
reported as key elements to a successful online learning experience (Song et al., 2004). 

In a more recent study, Ke and Xie (2009) reported that adults performed better in a structured online 
course where there were specifically designated areas for reading materials, lectures, assignments, sample papers, 
and evaluation rubrics. This aligns with work done by Tallent-Runnels et al. (2006) who reported that the literature 
on distance education identified well structured content as a top priority for online courses. Despite the literature 
supporting the need for efficient and effective online course design, and how the QM standards are becoming more 
widely used to assess online course design (Bogle, Cook, Day, & Swan, 2009), there is less research about how the 
Quality Matters standards are rated and perceived by students in terms of their perception of success in online 
courses. Nath and Ralston-Berg (2008) in a paper presented at the annual Conference of Distance Teaching and 
Learning at the University of Wisconsin, Madison, reported that participants in their study found all of the QM 
course design standards to be of value. Their study examined students’ perceived value of the QM standards using 
one-sample t-tests to compare mean scores followed by a correlation analysis to look at the relationship between 
student rankings of QM standards and their satisfaction with the online course. This study examined students’ 
perception of the elements of the Quality Matters standards as they were related to their perception of success in the 
online course, their rank order of the QM standards, and investigated whether or not a student’s age, major, or 
number of online classes previously taken affected their perception of the QM standards. 

Methods 

Participants 

Participants were invited from two public universities; one mid size university located in the west with an 
approximate enrollment of 14,000 and rated as a doctoral and research university by the Carnegie Foundation; the 
other, a larger university located in the mid west, with an approximate enrollment of 24,400 and rated as a high 
research activity university by the Carnegie Foundation for the Advancement of Teaching (2004). Both 
undergraduate and graduate degree programs including doctoral degrees, online courses, and online degrees are 
available from these two institutions. Non-probability convenience sampling was used based on access to faculty 
who were teaching online courses at these universities in the spring of 20 10. Seven instructors teaching a total of 11 
online classes agreed to send the survey to their students. The online classes surveyed included nursing education, 
Spanish, early childhood special education, managing technological change, technology and learning, and computer 
applications in education. The survey link was emailed to approximately 187 students, with 77 students completing 
the survey for a response rate of 41%. According to Rea and Parker (2005) a response rate of 50% for analysis and 
reporting of findings can be considered adequate as long as the researcher believes the responses are representative 
of the sample (p. 11). Archer (2008) suggests that a response rate of 51% is appropriate for output or impact type of 
evaluation surveys. 

Optimal sample size was estimated to equal 126 based on Cohen’s formula, (Green, 1991, p. 504) with a 
moderate effect size (R 2 = .13) and 12 variables for a multiple regression analysis. The student sample was 
comprised of graduate and undergraduate students from a variety of subject areas such as nursing education, 
educational technology, Spanish, and special education, with a range of previous online course experience. Forty 
seven percent of the participants had taken over 4 online classes previous to this study, and over 54% of the 
participants were between the ages of 23 and 37. Females represented 72% of the participants and males the 
remaining 28%. 

Instrumentation 

Because the classes were fully online courses with no face to face components, an online survey offered 
convenience, efficiency, and cost effectiveness for securing student input, and surveys have considerable credibility 
because of their widespread acceptance and use in academia (Rea & Parker, 2005). An electronic survey is also a 
logical way to reach students who are already using technology (the online course) in their academic endeavors. 

The online survey was crafted from the original QM rubric but with the statements reworded in order to 
make them current and relevant to students and to remove some of the academic jargon. For example one of the QM 
standards states, “Instructions make clear how to get started and where to find various course components” 
(MarylandOnline, 2008, p. 2). This was reworded to, “Instructions for navigating the online course, such as how to 
get started and where to find various course components.” In another example, the QM standard stated, “Minimum 
student preparation, and, if applicable, prerequisite knowledge in the discipline are clearly stated” (p. 2). This was 
reworded to, “Prerequisite knowledge or skills necessary to be successful in the course are explained.” The 
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reworded statements were then reviewed for clarity and preservation of the intent of the original Quality Matters 
items by a QM representative from one of the participating universities. Additionally, to provide context for students 
when taking the survey, they were asked to rate the QM standard while considering how important that item was to 
them and their success in the online class. This directive appeared at the top of each new page of the survey. 

Participants were asked to choose from a five point ordinal scale which was derived from the three point 
scale that the Quality Matters rubric uses. The QM rubric uses three possible evaluative responses regarding the 
criticality of the standard to the course: important, very important, and essential. I converted the response items to a 
five point scale based on research conducted by Lozano, Garcia-Cueto, and Muniz, (2008) who posited that the 
optimal range of Likert-type responses should be between four and seven because with fewer than four responses 
reliability and validity decrease and with more than seven responses the gains are minimal. Allen and Seaman 
(2007) also suggested including at least five response categories. Therefore, students were asked to rate the items on 
a scale from one to five as not important, somewhat important, important, very important, or essential to their 
success in the online class. 

In revising the survey to be student appropriate, several QM standards were eliminated, such as, “The 
course incorporates ADA standards and reflects conformance with institutional policy regarding accessibility in 
online and hybrid courses” (MarylandOnline, 2008, p. 16), because students had no way of knowing whether or not 
this was true for their online class. These revisions resulted in a survey comprised of 35 statements, down from the 
original 40 Quality Matters standards. The other standards eliminated from the survey were considered irrelevant or 
unknown from the students’ perspective. 

Additionally, in the learner engagement category, one standard contained more than two items for 
evaluation. In order for this survey item to be less ambiguous, the learner engagement category was broken down 
resulting in two additional standards’ statements. For example, the QM statement, “Learning activities foster 
instructor-student, content-student, and if appropriate to the course, student-student interaction (MarylandOnline, 
2008, p. 10) was broken down into three less ambiguous statements: (a) learning activities support my interactions 
with other students, (b) learning activities support my interactions with my instructor, (c) learning activities support 
my interactions with course content. 

Cronbach’s alpha calculated for the modified Quality Matters survey scores was .83 in the current study. 
According to Gliem and Gliem (2003), the closer Cronbach’s alpha is to 1.0 the greater the internal consistency of 
the items in the scale, noting that an alpha of .8 is a reasonable goal (p. 87). Additionally, five participant 
characteristic questions were presented in order to gather relevant information about the sample, and two open ended 
questions were added to elicit student perceptions about other items they thought important to their success in online 
courses. 

Subscales as presented in the original QM rubric consisted of eight categories with associated standards. 
When re-working the standards into a student appropriate survey, every effort was made to preserve the eight 
original categories and their standards. Table 1 depicts the original eight QM categories (subscales) with the number 
of related standards alongside the number of revised survey standards, and Cronbach’s alpha of internal consistency 
for each revised subscale. According to Garson (2008), the “widely-accepted social science cut-off is that alpha 
should be .70 or higher for a set of items to be considered a scale, but some use .75 or .80 while others are as lenient 
as .60” (Measures of Internal Consistency, Cronbach’s alpha section). All statistical analyses were computed using 
SPSS/PASW version 17. 



Table 1 

Quality Matters Categories with Items, Revised Items, and Cronbach ’s Alpha 
Original QM Standard Category Original Number of Revised number of 

standard items standards for survey 



Course overview and introduction 7 

Learning objectives (competencies) 5 

Assessment and measurement 5 

Resources and materials 4 

Learner engagement 4 

Course technology 7 

Learner support 4 

Accessibility 4 



7 

3 

5 

4 

6 
4 
4 
2 



Cronbach’s Alpha 
for revised 

subscales 

.726 

.825 

.653 

.621 

.635 

.537 

.834 

.338 
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The factors (categories) of accessibility, course technology, and to a lesser degree, assessment, resources, and 
learner engagement all show low internal consistency which indicates a certain lack of consistency in responses to 
these items (scales). 

In order to provide validity evidence of the QM survey scores, a principal component analysis with oblique 
rotation was run on the rating scales. The Promax rotation was chosen to allow the factors to correlate. The factor 
solution was composed of 35 items grouped into eight factors that accounted for 67% of the variance in the survey; 
Table 2 presents the percent variance for each of the eight factors. 

Table 2 

Variance Explained by Each Factor 

Factor % of variance 



1 Course Overview 


23.94 


2 Learning objectives 


11.27 


3 Assessment/Measurement 


9.26 


4 Resources /Materials 


5.67 


5 Learner engagement 


5.05 


6 Course technology 


4.34 


7 Learner support 


4.13 


8 Accessibility 


3.44 



Not all survey statements loaded on the eight Quality Matters designated categories. Factor one, had 66% 
of the questions from the QM category learner support, while factor two had 66% of items from the QM category 
course technology. Factor three had 50% of its items loading from the QM category learning objectives, and 50% 
loading from the QM category course introduction. Factor four had 50% of its items loading from the QM category 
learner engagement, and 50% of its items loading from course introduction. Factor five had 60% of its items loading 
from the QM category assessment and measurement, and factor six had just two items load, one from learner 
engagement and one from course technology. Factor seven had 40% of its items loading from the QM category 
resources and materials, and factor eight had two items loading both from the course introduction category. 

Procedures 

A link to the survey was emailed to each of the seven instructors who were teaching a total of 1 1 online 
classes beginning the ninth week of a 16 week semester. The instructors disseminated the link in a manner 
convenient for them. Once selected, the active survey link took participants to the informed consent letter, where 
students clicked an “I agree” button acknowledging their consent to participate and thereby launching the survey. A 
two week time frame was provided for participants to complete the survey, with reminder emails sent out mid way 
through this time frame. 

Preliminary Data Analysis 

Data collected for student majors were open ended and categorical in nature. In order to investigate the 
relationship between major and student perception of the QM standards, 1 looked for general themes and repetition 
of key words such as special education, early childhood special education community health, instructional 
technology and nursing. The major themes were then coded numerically in order to examine the relationship 
between major and student QM ratings. This text analysis resulted in seven major categories from a total of 67 
students who reported their major. These categories were: education (including early childhood ed. special ed. and 
deaf ed.), community health, nursing, instructional technology, psychology, library science, and others. 

Data Analysis 

Descriptive statistics were computed to derive the mean and standard deviation for each of the eight 
original Quality Matters categories in order to begin addressing the first research question: how do students rank the 
QM standards in terms of the item’s importance to their success in online classes. Additional descriptive statistics 
were used to examine student rankings of each of the 35 standards within the eight QM categories. Following that, 
Pearson’s product moment correlations were computed across all eight of the QM categories to see whether or not 
the categories were related to each other. 
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To address the second research question regarding a relationship between age and student perception of the 
QM standards, Spearman’s Rho was used to look at the correlation between age and scores from each of the QM 
categories or factors. Spearman’s Rho was used again to examine the relationship between the number of online 
classes previously taken and scores from each of the QM categories in order to answer the question, does the 
number of online classes previously taken explain students’ perception of the QM standards. 

The fourth research question, is a student’s major related to their perception of the Quality Matters 
standards, was addressed by performing a MANOVA to see the main and interaction effects of major (categorical 
variable), age range, and number of online classes previously taken on scores from each of the QM categories 
(multiple dependent interval variables). 

Open ended responses provided insights into participants’ views of online course elements and their related 
experiences. Common themes and text patterns were examined and then coded based on similarities and frequencies 
for interpretation in the context of the research questions. 

Results 

To examine the relationship of each of the eight QM categories with each other, Pearson’s product moment 
correlations were computed. Moderate to strong positive correlations were observed between course introduction 
and learning objectives, course introduction and engagement, course introduction and support, 
assessment/measurement and resources/materials, assessment/measurement and course introductions, and 
engagement and support. Weaker positive correlations were observed between course introduction and course 
technology, learning objectives and support, and learning objectives and accessibility, as well as between course 
technology and support, course technology and accessibility, resource/materials and support, and resource/materials 
and accessibility. Of note, no significant correlations were observed between accessibility and 
assessment/measurement, and between accessibility and resources/materials. 

Salient to answering the first research question is how students rank ordered each of the Quality Matters 
statements within the 8 categories. Of note, participants in this study only rated nine items as essential out of a total 
of the 17 rated essential by Quality Matters; these top nine items along with the Quality Matters value for the 
equivalent or similar item are presented in Table 3. 

Table 3 

Items Ranked as “ Essential ” by Students in QM Survey Compared to QM Ranking 



Item 


N 


% 


QM Value 


4.4 Instructional materials are easily accessible and useable. 


49 


60.49 


* 


3.2 It is clear how my work will be graded and how my participation will 
be graded. 


48 


60.00 


Essential 


6.2 Navigation throughout the entire course is logical, consistent and easy 
to follow, 


45 


56.25 


Essential 


1.1 Instructions for navigating the online course, such as how to get 
started and where to find various course components. 


44 


51.76 


Essential 


6.3 Technologies required for this course are either provided or easily 
downloadable 


43 


54.43 


Very Important 


2.3 The learning outcomes and objectives are realistic and achievable for 
the level of this course. 


39 


48.15 


Essential 


6.4 Instructions on how to access resources online are sufficient and easy 
to understand 


36 


45.00 


Very Important 


4.3 Instructional materials are sufficient for me to learn the subject. 


35 


44.30 


Very Important 


1.7 Minimum technology requirements are clearly stated. 
* There is no single similar item in the QM standards 


32 


37.21 


* 



Research question two examined the relationship between age and student perception of the QM items. 
Spearman’s rank correlation coefficient (r s ) was computed for age and each of the eight QM categories. Learning 
objectives had a moderate (Cohen, 1992), positive correlation with age, r s = .325, p = .004 and resources/materials, 
also had a moderate positive correlation with age, r s = .305, = .007 (significant at the 0.01 level, 2-tailed). Cohen 
(1992) considers r at .30 a medium effect size, and .5 to be a large effect (p. 157). However, according to Ferguson 
(2009) r at .5 is considered a moderate effect while .8 is considered a strong effect (p. 533). 
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The third research question investigated the extent to which the number of online classes previously taken 
by students explained their perception of the Quality Matters items. Again Spearman’s rank correlation coefficient 
(r s ) was computed for each of the eight QM categories and number of online classes previously taken. Only a small 
negative correlation (Cohen, 1992) was observed between accessibility and number of online classes previously 
taken, r s = -.226, p = .045 (significant at the 0.05 level, 2-tailed). Ferguson (2009) considers r at .2 to be a 
“recommended minimum effect size representing a ‘practically’ significant effect for social science data” (p. 533). 

Prior to answering the question about whether a student’s major was related to their perception of the 
Quality Matters standards, the data regarding majors was examined for reoccurring patterns and similarity, then 
similar majors were combined and assigned a numeric value. The most common major reported was instructional 
technology ( n = 24). 

To address whether a student’s major was related to his or her perception of the Quality Matters standards 
(research question four), a multivariate analysis of variance was calculated. A one-way MANOVA revealed no 
significant multivariate main effect for major; Wilks’ A = .432, F(48, 264.85) = 1 .028, p = .431, partial eta squared 
= .131. According to Ferguson (2009) a partial eta squared of .04 is a recommended minimum effect size that offers 
practical significance for social science data (p. 533), so the .13 found in this study represents a small practical effect 
of major on the QM standards. 

The open ended questions addressed student perceptions regarding what they felt were important elements 
in an online class as well as what other things they believed contributed to their success in an online class. These 
responses were examined for similar themes and patterns and were grouped into the following categories: instructor 
involvement, communication and feedback, and course design. 

Discussion 

The primary reason for this study was to investigate how students rank ordered elements of the Quality 
Matters standards compared to how the elements were ranked by the Quality Matters organization. Of the nine 
elements rated as essential by students. Quality Matters rated four of those items as essential too; they are: 1) clarity 
about how students will be graded (assignment and participation); 2) logical and consistent navigation; 3) 
instructions for where to find course materials; 4) realistic and achievable learning objectives. Interestingly, students 
also rated as essential the following five elements: 1) accessibility and usability of the materials, 2) technologies 
required for the course be provided or easily downloaded 3) instructions for accessing resources online be sufficient 
and easy to understand 4) instructional materials be sufficient for learning the subject 5) minimum technology 
requirements are clearly stated. The QM standards rank the following as very important, instead of essential 1) 
technologies required for the course be provided, 2) instructions for accessing resources online be sufficient, and 3) 
instructional materials be sufficient for learning the subject. There were no direct corresponding QM elements for 
the survey statements about accessibility and usability of the instructional materials, and, minimum technology 
requirements clearly stated. 

In examining the nine statements ranked as essential by students, a pattern emerges that is not specifically 
reflected in the QM categorization of these elements. The pattern appears to be one of students expressing the need 
for clear instructional guidance in order to be successful in the course. Guidance for finding their way around in the 
online class, for accessing and using online resources, guidance regarding the technical requirements for the class in 
addition to guidance about how to access and use the technologies that may be required for the online class. These 
findings offer practical suggestions about where instructors and course designers should focus their time, direction, 
and effort when building and fine tuning their online courses. Because these participants have extensive online 
experience (75 % had taken 3 or more online classes previously) it implies a certain familiarity with the online 
learning environment and an understanding about what it takes to be successful in that environment. This expertise 
may compensate for courses that are less well organized and less directive because experienced student may know 
the ropes and need less guidance. As Shea and Bidjerano (2008) reported, .more exposure to online environments 
may lead to a greater ability to integrate course materials, experiences, and interaction with others as an effective 
whole” (p. 356). 

On the other end of the rankings (not important to somewhat important), 38% of the students ranked the 
provision of netiquette or guidelines for how to behave in the online environment as either not important or 
somewhat important, while Quality Matters ranked netiquette as an important element. Based on the online 
experience level of this sample, it is reasonable to posit that they are familiar with online courtesies and appropriate 
online discussion participation, and do not perceive online etiquette as important to their success in the class. As one 
student wrote, “I am less concerned about the ‘netiquette’ and discussion board guidelines than I am about the 
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assessments in a particular course. By now I should know how to behave in the online environment, and most of my 
courses have involved the group self-policing to a moderate extent.” 

With current literature discussing the importance of creating a sense of community in the online 
environment, and introductions being one of the primary factors contributing to a sense of belonging, it is notable 
that 43% of the participants rated student self introductions in the class as not important to somewhat important to 
their sense of success in the class. In contrast, various studies point to online communities as contributing to student 
satisfaction (Shea, Li, Swan & Pickett 2005, Shea & Bidjerano, 2008, Song, Singleton, Hill & Koh, 2004 ) and an 
important element of community is students knowing each other virtually (social presence). Quality Matters rated 
student introductions as important. 

Consistent with rating student introductions low, 38% of the participants rated learning activities that 
supported interactions with other student s as not important to somewhat important, while QM rated this as 
essential. This finding is similar to Reisetter’s and Boris’ (2004) study which found that students placed a relatively 
low value on their interactions with peers, which again contrasts to current thought that student to student 
engagement is part of an efficient and effective learning environment. 

In examining the relationship between age and each of the QM categories, only a moderate positive 
association was found between age and two QM categories: learning objectives, and resources/materials. Learning 
objectives included items addressing clear explanation of learning outcomes, easy to understand instructions about 
achieving course goals and the alignment of objectives with course work. As students mature, they may become 
more self aware about what contributes to their success online and better understand how learning objectives, when 
clearly written and evident, can contribute to their success. 

The QM category for resources/materials depicts how materials support the stated learning objectives and 
that the materials need to have sufficient breadth and depth for students to learn. Similar to age and learning 
objectives, as students mature they may develop a more in-depth appreciation for the resources and course materials 
that are necessary for their success in the online class and avail themselves of these resources more frequently then 
what might be found in younger students. Similar to the results here, Ke and Xie (2009) found that age itself was not 
a predictor of adult students' learning satisfaction and performance. 

Surprisingly, the relationship between a student’s previous online course experiences and their perceptions 
of the QM categories was almost non-existent. Only a small negative correlation was observed between accessibility 
and number of online classes previously taken. This most likely indicates that student new to the online learning 
environment or students with less online experience present a greater need for links and headings that are self 
describing and meaningful, as well as needing equivalent audio or visual content in lieu of the text-intensive content 
found in most online learning environments. 

The two open ended questions were designed to elicit student input about other elements in an online 
course that contribute to their success as well as elements that make for a quality online course. After reviewing the 
student statements, three themes emerged: instructor involvement, instructor communication and feedback, and 
course design. Many students mentioned that instructor communication was important, and that the instructor be 
accessible, engaged, and provide timely feedback, although this was outside of the Quality Matters rubric. 
Additionally students wrote that they wanted interaction with their instructor and that the instructor needed to be 
organized. One student summed it up by writing, 

I can’t mention enough the need for timely, thoughtful feedback. I’ve had experiences where the instructor has not 
stuck to the published feedback schedule, and it makes it difficult to gauge whether or not I’m meeting the course 
goals. I don’t think there can be too much communication. I’ve not yet gotten sick of seeing an email or posting 
from an instructor. 

A second student commented about the need for regular feedback by writing, “. . .if I don’t hear that I am doing well 
or need to work to improve specific things, [then] what am I doing all of this work for?” Another wrote, “We might 
see a grade but comments on the grade are essential since you can’t speak with the professor one on one.” It is 
interesting to note that although the Quality Matters rubric and subsequent survey were directed towards gathering 
student perceptions about essential course design elements, and not on content delivery or instructor presence, a 
majority of students who wrote comments made mention of the instructor’s role and its criticality to their success 
online. 

Student comments about online course design further support the emphasis and importance placed by the 
QM standards on clear and easy navigation, and organization of content. As one student wrote, “It needs to be 
organized. Requirements, due dates, and activities need to be organized into sessions or modules and requirements 
for each session/ module need to be laid out.” Other students mentioned that having most of the content in one or 
two easily accessible places would contribute to their success, while another mentioned simple steps to obtain and 
submit assignments and consistent use of the learning platform as key to his or her success. One student aptly 
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summed it up, “The biggest thing to me is that the materials are easily accessible and it is not hard to find what you 
need for a task or assignment.” 

Of particular note is despite the results showing emphasis on learning goals and objectives and the need for 
objectives to align with tasks and assessments, (79% of participants rated clear objectives as very important to 
essential, 89% rated instructions about how to meet learning objectives as very important to essential, while both are 
rated as essential by QM), there were no student comments about objectives as a specific element contributing to 
their success in an online class. This may be because participants perceive objectives in a more transparent manner 
and assume if they know where to go, what to do, and what is expected of them, then objectives become an intrinsic 
part of that and do not need to be called out. This has implications for future research around the role that course 
objectives play in student success and satisfaction in online learning environments; perhaps explicit learning 
objectives are not necessary if course expectations are clear and relevant. 

Limitations 

Several limitations of this study restrict its generalizability to other student populations. First, there was no 
randomization of the sample, and secondly the sample size itself was not sufficient to provide adequate power for 
the statistical analysis. A few of the survey statements used “and/or” language which can cause confusion regarding 
how to respond to the statement (Rea & Parker, 2005). Also two survey statements about technology were 
confusing; it was not clear whether they were asking about the students’ prerequisite technology skills, or the 
technology requirements for participating in and accessing an online class. Also, placement of the survey statements 
in the prescribed QM categories should be reconsidered in light of the results of the exploratory factor analysis; 
many of the survey statements did not adhere to the QM prescribed categories. This calls for a re-examination of the 
QM categories as being adequate factors or aggregate representations of the various Quality Matters standards. 

Conclusion 

Overall, the findings of this study lend support to the QM standards as evidenced by student perceptions of 
those standards relative to their success in an online course. There were a few areas where students did not rate 
particular elements as essential where Quality Matters did, and conversely, where students rated a few elements as 
essential and QM did not. Further study of the Survey Responses with Quality Matters Rankings table found in the 
appendix may provide additional insights into areas that students perceived as very important or essential to their 
success in an online class. 

The survey results can be used to inform and guide instructors’ online course development efforts. For 
example, if students perceive that course introductions are important but not essential to their success, then does it 
make sense to spend much time on introductory activities? Netiquette is another area where online instructors and 
faculty may be able to spend less time, although in this study most participants had extensive previous online 
experience which may negate their need for explicit norms regarding online behavior and participation. Conversely, 
since students perceived instructions about accessing resources online, and the materials provided being sufficient 
for them to learn the content, as essential, the QM organization may want to reevaluate why they rated those 
elements as very important instead of essential. 

Future studies should seek a larger sample size and a broader cross section of participants that is more 
inclusive of undergraduates, males and females, other majors, and students over the age of 40, to see if similar 
results are found. Additionally, researchers in future studies may want to examine instructor activities within the 
online course that support student success in conjunction with the course design elements offered by the Quality 
Matters standards. Exploring essential online instructor competencies and behaviors necessary to support student 
success in the online environment offers opportunities for the Quality Matters standards organization to provide 
additional synergistic guidelines for the enhancement of teaching and learning online. 
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Appendix 



Survey Responses with Quality Matters Rankings 

Survey Statement # respondents % QM 

Rating 

Course Overview and Introduction 

1.1 Instructions for navigating the online course, such as how to get started and where to find various course 
components. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



2 2.35% 

7 8.24% 

5 5.88% 

27 31.76% 

44 51.76% Essential 



1 .2 A statement that introduces me to the purpose of this course & its components. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



1 1.16% 

9 10.47% 

18 20.93% 

35 40.70% 

23 26.74% Essential 



1.3 “Netiquette” or etiquette guidelines for how to behave online regarding email, discussion posting and other 
forms of course communication. 



Not important 


9 


10.47% 




Somewhat important 


24 


27.91% 




Important 


23 


26.74% 


Important 


Very important 


19 


22.09% 




Essential 


11 


12.79% 




1.4 An introduction from the instructor telling us about himself or herself. 




Not important 


3 


3.57% 




Somewhat important 


18 


21.43% 




Important 


25 


29.76% 


Important 


Very important 


31 


36.90% 




Essential 


7 


8.33% 




1.5 All of the students are asked to introduce themselves to the class. 




Not important 


11 


12.79% 




Somewhat important 


26 


30.23% 




Important 


25 


29.07% 


Important 


Very important 


18 


20.93% 




Essential 


6 


6.98% 
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Survey Statement # respondents % QM 

Rating 

1.6 Prerequisite knowledge or skills necessary to be successful in the course are explained. 



Not important 


1 


1.16% 




Somewhat important 


7 


8.14% 




Important 


26 


30.23% 


Important 


Very important 


29 


33.72% 




Essential 


23 


26.74% 




1.7 Minimum technology requirements 


are clearly stated. 






Not important 


1 


1.16% 




Somewhat important 


4 


4.65% 




Important 


18 


20.93% 


Important 


Very important 


31 


36.05% 




Essential 


32 


37.21% 




Learning Objectives 








2.1 The course learning outcomes and objectives are presented and clearly stated. 




Not important 


1 


1.23% 




Somewhat important 


4 


4.94% 




Important 


12 


14.81% 




Very important 


40 


49.38% 




Essential 


24 


29.63% 


Essential 


2.2 Instructions about how I can meet the learning goals and objectives are easy to understand. 


Not important 


1 


1.22% 




Somewhat important 


1 


1.22% 




Important 


7 


8.54% 




Very important 


44 


53.66% 




Essential 


29 


35.37% 


Essential 


2.3 The learning outcomes and objectives are realistic and achievable for the level of this course. 


Not important 


1 


1.23% 




Somewhat important 


0 


0.00% 




Important 


7 


8.64% 




Very important 


34 


41.98% 


Very important 


Essential 


39 


48.15% 





Assessment & Measurement 

3.1 The types of assessments and work given in this course are consistent with the stated learning objectives. 



Not important 


0 


0.00% 




Somewhat important 


1 


1.23% 




Important 


5 


6.17% 




Very important 


46 


56.79% 




Essential 


29 


35.80% 


Essential 
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Survey Statement 



# respondents % 



QM 

Rating 



3.2 It is clear how my work will be graded and how my participation will be graded. 



Not important 


0 


0.00% 




Somewhat important 


1 


1.25% 




Important 


3 


3.75% 




Very important 


28 


35.00% 




Essential 


48 


60.00% 


Essential 


3.3 The types of assessments and work given in this course are 


consistent with what I am asked to do and consistent 


with the resources available to me. 








Not important 


0 


0.00% 




Somewhat important 


0 


0.00% 




Important 


4 


4.94% 




Very important 


42 


51.85% 




Essential 


35 


43.21% 


Essential 


3.4 There are a variety of assessments used in the course. 






Not important 


1 


1.23% 




Somewhat important 


14 


17.28% 




Important 


24 


29.63% 




Very important 


33 


40.74% 


Very important 


Essential 


9 


11.11% 




3.5 Self check or practice assignments are provided for 


me with timely feedback. 




Not important 


3 


3.75% 




Somewhat important 


9 


11.25% 




Important 


20 


25.00% 




Very important 


35 


43.75% 


Very important 


Essential 


13 


16.25% 




Resources & Materials 








4. 1 Instructional materials relate to the learning outcomes and i 


course objectives. 




Not important 


0 


0.00% 




Somewhat important 


3 


3.70% 




Important 


10 


12.35% 




Very important 


48 


59.26% 




Essential 


20 


24.69% 


Essential 


4.2 The relationship between the instructional materials and learning activities is 


clearly explained. 


Not important 


0 


0.00% 




Somewhat important 


3 


3.75% 




Important 


20 


25.00% 




Very important 


43 


53.75% 




Essential 


14 


17.50% 


Essential 
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Survey Statement 


# respondents % 


QM 

Rating 


4.3 Instructional materials are sufficient for me to learn the subject. 




Not important 


0 


0.00% 




Somewhat important 


0 


0.00% 




Important 


12 


15.19% 




Very important 


32 


40.51% 


Very important 


Essential 


35 


44.30% 





4.4 Instructional materials are easily accessible and useable. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



0 0 . 00 % 

0 0 . 00 % 

2 2.47% 

30 37.04% 

49 60.49% 



NA 



Learner Engagement 

5.1 What I am asked to do (learning activities) promotes the achievement of the learning outcomes and course 
objectives. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



1 1 . 22 % 

1 1 . 22 % 

5 6.10% 

54 65.85% 

21 25.61% Essential 



5.2 Learning activities support my interactions with other students. 



Not important 


12 


14.63% 


Somewhat important 


19 


23.17% 


Important 


23 


28.05% 


Very important 


23 


28.05% 


Essential 


5 


6.10% 



Essential 



5.3 Learning activities support my interactions with my instructor. 



Not important 


3 


3.66% 


Somewhat important 


14 


17.07% 


Important 


18 


21.95% 


Very important 


35 


42.68% 


Essential 


12 


14.63% 



Essential 



5.4 Learning activities support my interaction with the course content. 



Not important 


1 


1.23% 


Somewhat important 


1 


1.23% 


Important 


10 


12.35% 


Very important 


47 


58.02% 


Essential 


22 


27.16% 



Essential 
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Survey Statement 



# respondents % 



QM 

Rating 



5.5 The requirements for interaction in this course are clearly explained. 



Not important 


0 


0.00% 




Somewhat important 


1 


1.25% 




Important 


8 


10.00% 




Very important 


40 


50.00% 


Very important 


Essential 


31 


38.75% 





5.6 Clear standards are set for instructor response time and instructor availability (turn-around time for emails, grade 
postings, etc.). 



Not important 


0 


0.00% 


Somewhat important 


5 


6.10% 


Important 


18 


21.95% 


Very important 


30 


36.59% 


Essential 


29 


35.37% 



Very important 



Course Technology 

6. 1 The tools and media in the online course support my becoming an active learner. 



Not important 


2 


2.50% 


Somewhat important 


1 


1.25% 


Important 


25 


31.25% 


Very important 


32 


40.00% 


Essential 


20 


25.00% 



Essential 



6.2 Navigation throughout the online course is logical, consistent and easy to follow. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



0 0 . 00 % 

0 0 . 00 % 

10 12.50% 

25 31.25% 

45 56.25% Essential 



6.3 Technologies required for this course are either provided or easily downloadable. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



0 0 . 00 % 

0 0 . 00 % 

4 5.06% 

32 40.51% 

43 54.43% 



Very important 



6.4 Instructions on how to access resources online are sufficient and easy to understand. 



Not important 
Somewhat important 
Important 
Very important 
Essential 



0 0 . 00 % 

0 0 . 00 % 

10 12.50% Important 

34 42.50% 

36 45.00% 
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Learner Support 

Survey Statement 



# respondents % 



QM 
Rating 

7. 1 Course instructions explain or provide a link to clear descriptions of the technical support offered. 



Not important 


0 


0.00% 




Somewhat important 


4 


5.00% 




Important 


20 


25.00% 




Very important 


33 


41.25% 


Very important 


Essential 


23 


28.75% 





7.2 Course instructions explain or provide a link to clear explanations about how UNC’s academic support system 
can assist students in effectively using the resources provided. 



Not important 


4 


5.00% 




Somewhat important 


8 


10.00% 




Important 


32 


40.00% 


Important 


Very important 


21 


26.25% 




Essential 


15 


18.75% 





7.3 Course includes or links to a clear description of the academic support offered to students (such as advising, 
financial aid, the library etc.). 



Not important 


10 


12.50% 




Somewhat important 


16 


20.00% 




Important 


25 


31.25% 




Very important 


21 


26.25% 


Very important 


Essential 


8 


10.00% 





7.4 Course includes, or links to tutorials and resources that answer basic questions related to research, writing, 
technology, etc. 



Not important 


6 


7.50% 




Somewhat important 


9 


11.25% 




Important 


32 


40.00% 


Important 


Very important 


24 


30.00% 




Essential 


9 


11.25% 





Accessibility 

8.1 Course materials and online course pages provide equivalent alternatives to auditory and visual content. 



Not important 


7 


8.86% 




Somewhat important 


12 


15.19% 




Important 


21 


26.58% 




Very important 


27 


34.18% 


Very important 


Essential 


12 


15.19% 





8.2 Course pages have links and headings that are self-describing and meaningful. 



Not important 


0 


0.00% 




Somewhat important 


4 


5.00% 




Important 


14 


17.50% 




Very important 


41 


51.25% 


Very important 


Essential 


21 


26.25% 
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Learning Statistics with Cognitive and Metacognitive Strategies that Facilitate 
Multiple External Representations Integration 
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Abstract 



The literacy of multiple external representations (MERs) (e.g., graphs, and formulas) and the ability of 
integrating MERs are critical to learning in the STEM fields (Science, Technology, Engineering, and Mathematics). 
This research investigated instruction on cognitive and metacognitive strategies with potential in facilitating students 
to integrate MERs. This study experimentally compared the learning performance by the four groups (N = 118 
undergraduate students) differed regarding the instruction on strategies each group received. The results indicated no 
difference on introductory statistics learning performance among groups. Analysis showed there might be potential 
ceiling effect due to the specific sample's high mathematics aptitude. Namely, students with high aptitude relevant to 
the subject of learning might not benefit from the instruction on strategies. Interpretation of the results and 
implications for future research were discussed. 



Purposes 

Students in the Science, Technology, Engineering, and Mathematics (STEM) fields are confronted with 
multiple external representations (MERs) in their learning material. The ability to learn and communicate with these 
MERs not only has significant impact on their learning achievement, but also their future performance in workplace. 
Past research has indicated the integration of MERs is not automatic, and students need instructional support to 
master the learning and integration of MERs. While researchers have provided learning environments with pre- 
integrated MERs or interactive capability, results were mixed in terms of effectiveness of these environments. A few 
research programs have indicated that students need to be supported with instruction on strategies that help guide 
their cognitive and metacognitive processes during learning from MERs and integrating them, so they can engage in 
deeper learning. Thus, this study aims to investigate the effects of instruction on strategies, including a self- 
explanation strategy and an MER integration strategy, on students’ learning outcomes on statistics, a domain with 
extensive use of MERs. 



Theoretical Framework 

Multiple external representations refer to instructional materials that contain two or more separate 
representations, while integration of MERs refers to the cognitive processes in which the learner actively engaged in 
understanding, connecting, and aligning two or more external representations that were once separate (Van Meter & 
Garner, 2005; de Jong et al., 1998). In MER-rich materials, MERs serve different functions depending on their 
characteristics. Ainsworth (1999) proposed the functional taxonomy of multiple external representations for 
analyzing the benefits of learning with MERs. She argued that there are three different beneficial functions of MERs 
when involved in learning, including providing complementary information, constraining interpretation, and 
engaging in deeper learning. 

Whereas Ainsworth’s taxonomy describes the potential functions of MERs, Mayer has proposed a model to 
explain people’s cognitive processes of integrating MERs, the Cognitive Theory of Multimedia Learning (CTML) 
(Mayer, 2005). This theory suggests that learning in the multimedia environment involves the following cognitive 
processes either linearly or nonlinearly: 1) selecting relevant words for processing in verbal working memory, 2) 
selecting relevant images for processing in visual working memory, 3) organizing selected words into a verbal 
model, 4) organizing selected images into a pictorial model, 5) integrating the verbal and pictorial representations 
with each other and with prior knowledge. Due to limited capacity of working memory, people need to select 
relevant information to process. Because of dual channel characteristic, the selected information needs to be 
processed and organized separately and then finally integrated. 
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Instructional Support for Cognitive and Metacognitive Processes While Integrating MERs 



Past research presented mixed results of the effectiveness of instructional support in helping learners 
integrate across MERs. Research studies showed promising instructional intervention implemented in pre-fabricated 
learning environments, such as pre-integrated MERs (Bodemer, Plotetzner, Feuerlein, & Spada, 2004), drag-and- 
drop technique (Bodemer & Faust, 2006), and interactive hyperlinks (Seufert & Bmnken, 2006). However, the same 
intervention also led to different results for high school students and college students (e.g., Seufert & Briinken, 
2006). It seems that simply the existence of MERs or even an environment with affordance to help integrate MERs 
do not necessarily lead to MER integration and better learning. 

Learning with MERs and integrating them involve one’s cognition and metacognition. Due to limited 
cognitive capacity, when processing MERs individuals are forced to decide about which piece incoming information 
to attend to, select, organize, and connect. This decision-making process requires one’s metacognitive strategies, 
which are the “techniques for allocating, monitoring, coordinating, and adjusting these limited cognitive resources” 
(Mayer, 2005, p. 36). Mayer’s claim and the support from past research indicates that there is a need to support 
learners’ cognitive processes as well as conscious control of their cognitive activity during the learning tasks, so 
their learning with MERs can improve. 

Self-explanation (SE) as a Strategy for Integrating MERs 

Self-explanation (SE) refers to a learning technique where individuals explain what they learn to 
themselves. It helps engage students in active learning, attend to learning material, and effectively monitor their own 
evolving understanding. The major cognitive and metacognitive processes involved in SE include generating 
inferences to fill in missing information, integrating information within the study materials, integrating new 
information with prior knowledge, and monitoring as well as repairing faulty knowledge (Roy & Chi, 2005). 

Past research indicated that SE has had positive correlation with learning achievement. For example, Renkl 
(1997) indicated that self-explanation of solution steps in worked examples were good predictors of learning 
outcomes. In addition, SE has been established as a useful and teachable technique for enhancing learning across 
different subject areas, such as human circulatory systems (Chi, de Leeuw, Chiu, & Lavancher, 1994), basic 
economics (Renkl, Stark, Gruber, & Mandl,1998), and argumentation skills (Schworm & Renkl, 2007). More 
research is needed to determine if SE would remain an effective technique if applied to scaffold MER integration. 

Guided by the review of literature, this research aims to answer the research question, “Would instruction 
on cognitive and metacognitive strategies, such as self-explanation and MER integration, make a difference on 
students’ learning performance in a STEM domain text with extensive use of multiple external representations?” 

Methods, Data Sources, and Materials 



Sample 



One hundred and eighteen (N= 118) undergraduate students were recruited from two advanced biology 
courses and one introductory educational psychology course in a northeastern public university in the U.S. to 
participate in this study. Among the 118 students, there were 55 males and 63 females, and 73% of them were at 
their 3rd or 4th year. The mean SAT Mathematics score was 664.38. The students had taken an average of 2 college- 
level mathematics courses. Most of the students have a major in Biology (50.85%), Premedicine (31.36%), or 
Elementary Education (7.63%). 

Procedure and Learning Environment 

In this research, all participants completed all the required units delivered with ANGEL, a web-based 
learning management system hosted on the university server. The required units included: 1) demographic survey, 2) 
instruction on strategies, 3) statistics learning material, and 4) posttest. Also, participants followed the prescribed 
sequence to linearly go through the four units, with the completion of one unit activating the next unit. 

Research Design and Treatments 

This research employed a one-factor design with four instructional conditions, including one control group 
and three experimental group: 1) no instruction on strategies (i.e., control group-C), 2) instruction on self- 
explanation strategies (SE), 3) instruction on MER integration strategies (MI), 4) combined instruction on self- 
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explanation strategies and MER integration strategies (SEMI). The independent variable is the instruction on 
learning strategies, while the dependent variable is students’ learning outcomes on introductory statistics content. 

The first experimental group received instruction on self-explanation strategies and two prompts to guide the self- 
explanation on the statistics material by typing in a text box. The second experimental group received instruction on 
MER integration strategies but was not provided with any prompts to guide explanations. Instead, this group was 
asked to type what they thought while studying the material. The third experimental group received a combined 
instruction on self-explanation strategies and MER integration strategies and also two prompts of explaining how the 
MERs in the material were related to each other. The control group received no instruction or prompts, but was still 
asked to type their thoughts while studying the material. 

Statistics Learning Material 

The learning material of introductory statistics was developed through adapting statistics content from 
multiple sources. The statistics content included a variety of MERs: 7 tables, 1 1 graphs, 5 formulas, and 4830 words. 
The content included five major sections: Frequency Distribution, Normal Distribution, Measure of Central 
Tendency, Variability, and Correlation. 

Instillments 

A posttest on introductory statistics was developed to assess students’ learning performance on the major 
concepts covered in the learning material. While it was designed to measure concept learning, effort was also made 
to ensure students’ need to integrate MERs to answer the questions. The MERs used in this test paralleled that of 
which the students were presented in the statistics learning material: text, graphs, tables, and formulas. The test 
included a total of 42 multiple-choice questions. 



Results and Findings 

The results indicated a high reliability (a = .775) for the scores obtained from the posttest measure. The 
correlation analysis indicated that students’ posttest scores were positively significantly correlated with their highest 
SAT Mathematics score (r = .50, p = .000), highest SAT Verbal (r = .47. p = .000), and self-reported cumulative 
GPA (r = ,18,/» = .048). The descriptive analysis of posttest scores and demographics by condition were presented 
in Table 1. 



Table 1. Mean Scores of Posttest and Demographics by Condition 







Condition 




C 

(n = 30) 


SE 

(n = 31) 


MI 

(« = 28) 


SEMI 
(. n = 29) 


Posttest 


33.67 (4.64) 


33.81 (4.93) 


34.07 (4.00) 


35 (3.82) 


SAT Math 


656.87 (94.87) 


654.00 (92.57) 


656.89 (93.55) 


690.48 (96.55) 


SAT Verbal 


593.20 (123.85) 


622.71 (105.79) 


610.82 (102.36) 


633.59 (102.05) 


GPA 


3.40 (0.39) 


3.50(0.31) 


3.54 (0.40) 


3.54 (0.23) 
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Control Group vs. Experimental Groups 

There were no significant differences in statistics posttest scores between the control group and 1) self- 
explanation group (t(59) = -.11,/;- .91); 2) the MER integration group (t(56) = -.36, p = .72); 3) the combined 
group (t (57) = -1.20,/; = .23). 

Experimental Group vs. Experimental Group 

There were no significant differences in statistics posttest scores between 1 ) the self-explanation group and 
the MER integration group (t(57) = -.23, p = .82); 2) the self-explanation group and the combined group (t( 56) = - 
1.04 ,p = .30); 3) the MER integration group and the combined group (7(55) = -.90 ,p = .37). 

Discussions 

The following were some possible explanations for the lack of effect of the treatments. First, there was a 
possible ceiling effect of students’ SAT Mathematics scores on students’ statistics posttest scores in this study. The 
participants’ mean SAT mathematics score was 664.38, which indicated the students overall had relatively high 
mathematics aptitudes. It is possible that the students with stronger mathematics background might not benefit as 
much from the instruction on strategies for learning the material highly-related to mathematics. For future research, 
we suggest to consider this aspect and recruit students with lower mathematics ability to examine if those students 
could benefit more from the instructions on strategies. Second, it is likely that the posttest appeared to be relatively 
easy for this particular group of participants even though the learning material might have posed some challenges. 
The posttest scores did not reflect the impact of the instruction on strategies. Again, this aspect needs to be examined 
when the posttest is administered to the students with lower mathematics ability. Third, the treatments might not 
have elicited the expected cognitive processes. That is, the self-explanation group generated more high-quality 
explanations, the MER integration group engaged in integrating MERs presented in the material, and the combined 
group performed both cognitive processes. This possibility would need to be confirmed (or rejected) through an in- 
depth analysis of students’ typed thoughts or self-explanations for the evidence of high-quality self-explanations and 
MER integration. The results, in turn, would have implications for revising the statistics learning materials and the 
instruction on strategies for future research. 



Significance 

This study aims to improve undergraduate students’ learning of introductory statistics, a foundational 
subject that provides the mathematical tool for understanding quantitative information in science disciplines such as 
biology and social sciences such as sociology, psychology, and education. Theoretically, this research seeks to 
advance the understanding of students’ use of cognitive and metacognitive strategies for learning with MERs. In 
practice, this research investigates the effects of instructions on strategies in helping students learn content that uses 
MERs extensively. In addition to experimenting and further understanding of how to support learning with MERs, 
this study contributes to the area of researching MER integration by developing a reliable statistics test that requires 
integrating MERs in the question stem and the corresponding choices in order to score. This format and 
development of this test can serve as a preliminary model of measurement development for future studies related to 
MER integration. The results of this study are expected to lead to valuable empirical results for researchers and 
educators in terms of exploring and improving instruction on strategy use in statistics as well as STEM domains that 
require MER integration in learning. 
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Abstract: This study was conducted to investigate the relationship between the usage and technology acceptance of 
Web 2.0 applications among college students. 449 individuals voluntarily participated in the study in 2009. The 
survey consisted of 95 items drawn from selected categories from the Unified Theory of Acceptance and Use of 
Technology (UTAUT). The results indicated significant relationships between the usage and technology acceptance 
on seven Web 2.0 applications (blog, wiki, instant messengers, online social community/Facebook, online video 
sharing/YouTube, online games, immersive virtual environment/Second Life). The study also found that students 
were more strongly influenced by perceptions of performance expectancy and effort expectancy on the use of blogs, 
wikis, and immersive virtual communities. These findings of the study imply that educators should take factors of 
technology acceptance into consideration when integrating Web 2.0 applications in the classroom settings. 

Introduction 

Web 2.0 applications such as blog, instant messenger, wiki, video sharing tool, and web conferencing tools 
are gaining popularity on college campuses. Students use them to create their own content on the web, contribute 
and collaborate with others, and develop social networks via multiple formats of media and representation (O’Reilly, 
2005). Although these activities suggest the possibility of using Web 2.0 applications for instructional purposes, 
they also raise the question of how students are motivated to use these web-based technologies (Lee, Cheung, & 
Chen, 2005). Instructors should take measures of user acceptance and usage into consideration when integrating 
Web 2.0 applications in the classrooms, which, however, are lacking in current literature. Therefore this study was 
designed as an effort to better understand how college students perceive and use Web 2.0 applications when asked in 
the context of completing learning tasks. Specifically we were interested in which technology acceptance factors 
most influence students’ utilization of Web 2.0 applications as an important first step for educators to design and 
develop an efficient learning environment (Curry, 1987; Marshall, 1987; Federico, 2000). This study intended to 
answer the following questions: 

• How do students accept different Web 2.0 applications? 

• What is the empirical relationship between the usage of Web 2.0 applications and technology 
acceptance? 

In the following sections we provide a brief overview of Web 2.0 applications and the measurement of 
technology acceptance, method of the study, and preliminary data analysis and results. Then we discuss the 
implications of our finding and how they inform the design of an efficient Web 2.0-enriched learning environments. 

Web 2.0 Applications for Learning 

Web 2.0 is a collective term describing a group of web-based technologies that broaden users’ 
communication capabilities and options (Anderson, 2007). Millions of web users are uploading contents on 
YouTube, and MySpace daily, which evidences a prominent growth of Web 2.0 applications in recent years (Time 
magazine, 2007). Timothy O’Reilly (2005), who initially created the term, defines Web 2.0 as an active and open 
web architecture that values users’ participation and contribution. McLoughlin & Lee (2007) suggested that Web 2.0 
is more personalized and interactive than the first generation of Web applications (i.e., the web was only used by 
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web users to retrieve information). Proactive participation, connectivity and collaboration are main features of Web 

2 . 0 . 



Amongst a variety of Web 2.0 applications, some do bear values in facilitating teaching and learning. Fors example, 
blogs are suggested to facilitate the publication of knowledge, reflection, and knowledge construction (Ferdig & 
Trammell, 2004). Student use them to build community, consolidate resources, share their ideas with their friends, 
and keep their personal journals (Kerawalla, Minocha, Kirkup, & Conole, 2008). Instant messenger is another tool 
that enables students to communicate in real time with two or more people based on texted messages (Saeed, Yang, 

& Sinnappan, 2009), which promotes collaborative learning, facilitates active interaction among team members, and 
encourages students to learn communication skills (Lu, Zhou, & Wong, 2009). Web 2.0 applications are also 
considered to be an efficient tool in conducting case studies due to their collaborative nature, which is based on an 
experiential learning approach (Fluang & Behara, 2007). Klamma, Chatti, Duval, Hummel, Hvannberg, Kravcik, 
Law, Naeve, & Scott (2007) further suggested that Web 2.0 applications have great potential to facilitate and 
enhance lifelong learning by connecting students in collaborative environments with diminishing boundaries. There 
is growing evidence that many people are engaged in a wide range of technologies-based informal learning at home 
and in their community (Selwyn, 2007). Web 2.0 seems to have substantial possibilities for students who have 
various needs to enhance their learning experiences through enriched interactions (Bryant, 2006; McLoughlin & Lee, 
2007). Safran, Helic, & Gutl (2007) emphasized the potential of Web 2.0 applications because they make it possible 
to uphold critical and analytical thinking, facilitate intuitive and associational thinking, support analogical thinking 
through easy access to rich information and allow the interaction of various opinions in a learning environment. 

In short, Web 2.0 applications have strong possibilities to be utilized in class settings for students’ learning. 
Educators have to consider how to integrate these unique features of Web 2.0 applications to accommodate the 
different factors influencing their students’ technology usage. Understanding the factors influencing the utilization 
of technology by students is important for designing and developing effective web based learning environments (Lee, 
Cheung, & Chen, 2005). Therefore, exploring various motivators affecting the usage and acceptance of Web 2.0 
applications for college students might provide useful insights for their integration in formal instructional settings. 

Technology acceptance and Web 2.0 applications 

Technology acceptance model (TAM) has been used to measure how users accept particular technologies. 

It was introduced by Davis (1968) to explain computer usage behavior. Since then, this model has been utilized to 
understand the acceptance of information technology in extensive empirical research (Venkatesh, Morris, Davis, & 
Davis, 2003). 

The TAM provides a basis for discovering the impact of external variables on internal beliefs, attitudes, and 
intentions (Marchewka, Liu, & Kostiwa, 2007). The model assumes that beliefs about usefulness and ease of use are 
always the primary determinants of information technologies adopted in organizations. Davis (1989) explains that 
these two determinants serve as the foundation for attitudes toward using a particular system, which in turn 
determines the intention to use, and then generates the actual usage behavior. TAM defines the perceived usefulness 
as the extent to which a person believes that using a system would enhance his or her performance. On the other 
hand, perceived ease of use refers to the extent to which a person believes that using a system would be free of 
mental effort (Davis, 1989). Prior research has implemented TAM in a variety of computer-based environments. 
Earlier studies have explored TAM’s applications in testing user acceptance of information technology, such as 
word processors (Davis et al., 1989), spreadsheet applications (Mathieson, 1991), e-mail (Szajna, 1996), web 
browsers (Morris & Dillon, 1997), websites (Koufaris, 2002), and Blackboard. Recently the model has been 
implemented in studies related to e-learning (Landry, Griffeth & Hartman, 2006; Masrom, 2007; Selim, 2003). 

The Unified Theory of Acceptance and Use of Technology (UTAUT)(Venkatesh et al. 2003), a more recent 
instrument, has synthesized existing models related to the concept of technology acceptance (Oshlyansky, Cairns, & 
Thimbleby, 2007). The UTAUT consists of eight constructs: performance expectancy, effort expectancy, social 
influence, facilitator conditions, self-efficacy, anxiety, behavioral intention to use, and attitude toward using 
technology, which may collectively influence the usage behavior of the technology. 

The UTAUT model has been applied to examine research that is related to academic settings and the 
workplace. It has been used to evaluate the acceptance of Web 2.0 applications (Huang, Yoo, & Choi, 2008), virtual 
learning environment (e.g. by Sumak, Polancic, & Heircko, 2010), podcasting (e.g. by Ho, Chou, & O’Neill, 2010), 
and even microblogging user acceptance by Meyer & Dibbern (2010). 
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Method 



The study was conducted through an online survey in 2009 in a public Midwestern university in the U.S. 
targeting seven Web 2.0 applications. Correlation analyses were conducted to address the aforementioned research 
questions. Some data were deleted because of errors or incompletion, which resulted in the following datasets for 
analysis: 110 sets for blogs, 160 for wikis, 88 for instant messengers, 100 for social networking tools, 108 for online 
video sharing, 169 for online game, and 244 for immersive virtual environments. All data were collected via 
voluntary participations. 

Measuring Utilization Level of Web 2.0 

This research team targeted seven Web 2.0 applications. Blog, wikis, instant messengers, online social 
networking (eg., Facebook), online video sharing (eg., YouTube), online games, and immersive virtual environment 
(eg., Second Life) are included in this study. To investigate the utilization level of Web 2.0 applications, the 
research team created surveys based on selected items from the Unified Theory’ of Acceptance and Use of 
Technology’ (UTAUT) (Venkatesh, Morris, Davis, & Davis, 2003) by using a 9-point Likert Scale in the following 
categories: Performance expectancy, effort expectancy, attitude, social influence, and anxiety. See Table 1 for 
survey. UTAUT is the synthesis of eight other models (ie., the theory of reasoned action, technology acceptance 
model, motivational model, theory of planned behavior, model of PC utilization, innovation diffusion model, and the 
social cognitive theory), which measures a unified technology acceptance rate expressed by individuals or 
organizations. 



Table 1. Acceptance of Web 2,0 applications 



Categorization 


Questions 


Code 


Usage of Web 2.0 


• 1, Flow often do you use it? 


1 


applications 


• 2, Flow much time do you usually spend each time you 


(frequencies and time) 


use it 


z, 




• 1,1 would find it useful in my learning tasks 


3 


Performance expectancy 


• 2, Using it enables me to accomplish tasks more 

quickly 


4 

5 




• 3, Using it increases my productivity 




Effort expectancy 


• 4, Learning to use it is easy for me 


6 




• 5, Using it for learning is a good idea 


7 


Attitude toward using Web 


• 6, It makes learning more interesting 


8 


2.0 applications 


• 7, Learning with it is fun 


9 




• 8, 1 like learning with it 


10 


Social influence 


• 9, People who influence my behavior think that I 

should use it 


11 


Anxiety 


• 10, 1 feel apprehensive about using it 


12 


• 1 1, It is somewhat intimidating to me 


13 



Data Analysis and Results 



Data Analysis 

Based on the research questions, this study used both descriptive and inferential statistics for data analysis. 
First, the research team used descriptive statistics to report the distribution of the usage of technology and Web 2.0 
utilization levels. Then inferential statistics (ie., correlation analysis) was conducted to identify the relationships 
between participants’ usage and Web 2.0 acceptance. 
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Participants 

The study collected responses from 449 undergraduate students in a public Midwestern university. Of the 
1 10 completed surveys for blogs, 36 of them were completed by males (33 %) and 74 by females (67 %). 39(35.5%) 
participants were freshmen, 42 (38.1%) sophomores, 21(19.1%) juniors, 4(3.6%) seniors, and 1(0.9%) was a 
graduate student. For wiki, a total of 160 datasets were analyzed. 46(28.7%) of them were male and 1 14(71.3%) 
were female. For instant messenger, 31 of the 88 were male (35.2 %) and 57 were female (64.8 %). 33(37.5%) 
participants were freshmen, 33 (37.5%) sophomores, 15(17.0%) juniors, 3(3.4%) seniors, and 1(1.1%) was a 
graduate students. For social networking tools, a total of 100 datasets were analyzed. 32(32.0%) of them were 
freshmen, 38 (38%) sophomores, 22 (22%) juniors, 4(4%) seniors, and 1(1%) was a graduate students. 36 of 100 
participants were male and 64 were female. A total of 108 dataset were analyzed for online video sharing tools. 
47(43.5%) students were freshmen, 37(34.2%) sophomores, 17(15.7) juniors, 3(2.7%) seniors, and 2(1.85%) were 
graduate students. For online game, a total of 160 datasets were analyzed. 65(38.5%) of them were freshmen, 
65(38.5%) sophomores, 28(16.6%) juniors, 7(4.14%) seniors, and 2(1.18%) were graduate students. For immersive 
virtual environment tools, a total of 244 datasets were analyzed. 78(32%) of them were male and 166(68%) were 
female. 93(38.1%) were freshmen, 95(38.9%) sophomores, 43(17.6%) juniors, 9(3.68%) seniors and 2 (0.8%) were 
graduate students. 

Table 2, Academic Year of Participants 



Frequency (percent %) 



X. Web 2.0 
\ applic 
X^tions 

Academic yeab\ 


Blog 


Wiki 


Instant Messenger 


Social 

Networking 


Online Video 
Sharing 


Online 

Game 


Immersive 

virtual 

environments 


Freshman 


39(35.5) 


66(41.3) 


33(37.5) 


32(32.0) 


47(43.5) 


65(38.5) 


93(38.1) 


Sophomore 


42(38.2) 


59(36.9) 


33(37.5) 


38(38.0) 


37(34.3) 


65(38.5) 


95(38.9) 


Junior 


21(19.1) 


27(16.9) 


15(17.0) 


22(22.0) 


17(15.7) 


28(16.6) 


43(17.6) 


Senior 


4(3.6) 


6(3.8) 


3(3.4) 


4(4.0) 


3(2.8) 


7(4.1) 


9(3.7) 


Graduate 


1(0.9) 


1(0.6) 


1(1.1) 


1(1.0) 


2(1.9) 


2(1.2) 


2(0.8) 


Others 


3(2.7) 


1(0.6) 


3(3.4) 


3(3.0) 


2(1.9) 


2(1.2) 


2(0.8) 


Total 


110(100) 


160(100) 


88(100) 


100(100) 


108(100) 


169 


244 



Table 3. Academic major of study participants 





Frequency(percent) 


\ Web 
2dk 

applications 
Major \ 


Blog 


Wiki 


Instant Messenger 


Social Online Video 

Networking Sharing 


Online 

Game 


Immersive 

virtual 

environments 


Education 


68(61.8) 


103(64.4) 


54(61.4) 


61(61.0) 


71(65.7) 


109(64.5) 


153(62.7) 


Business 


1(0.9) 


1(0.6) 


0(0.0) 


1(1.0) 


1(0.9) 


0(0.0) 


2(0.8) 


Engineering 


1(0.9) 


2(1.3) 


2(2.3) 


1(1.0) 


0(0.0) 


2(1.2) 


2(0.8) 


Science 


10(9.1) 


13(8.1) 


7(8.0) 


6(6.0) 


6(0.9) 


14(8.3) 


21(8.6) 


Liberal Arts 


21(19.1) 


30(18.8) 


17(19.3) 


22(22.0) 


21(19.4) 


28(16.6) 


43(17.6) 


Others 


9(8.2) 


11(6.9) 


8(9.1) 


9(9.0) 


9(8.3) 


16(9.5) 


23(9.4) 


Total 


110 


160 


88 


100 


108 


169 


244 



The respondents were also asked to indicate their major. See Table 3 for participants’ academic major. A 
majority of the respondents are majoring in education. 
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Table 4 Gender of Participants 





Frequency (Percent %) 


Web 
2. applications 

Gender \ 


Blog 


Wiki 


Instant Messenger 


Social 

Networking 


Online Video 
Sharing 


Online 

Game 


Immersive 

virtual 

environments 


Male 


36(32.7) 


46(28.8) 


31(35.2) 


36(36.0) 


37(34.3) 


53(31.4) 


78(32.0) 


Female 


74(67.3) 


114(71.3) 


57(64.8) 


64(64.0) 


71(65.7) 


116(68.6) 


166(68.0) 


Total 


110 


160 


88 


100 


108 


169 


244 



Descriptive Statistics Analysis 

The usage frequency of Web 2.0 applications was identified (see Table 4). When asked in the context of 
completing formal learning tasks, 138(86.3%) students responded that they have never used a wiki and 126(74.6%) 
students never used online games, 239(98%) of 244 total participants responded that they have never used a social 
virtual environment which indicated that social virtual environments are most unfamiliar to students. On the 
contrary, 31(35.2%) of 88 students responded that they used instant messengers more than 5 times per week. Online 
social networking sites were also most familiar to participants. 



Table 5 Analysis of frequency of Web 2.0 applications per week 


Web 2.0 applications 


I have 
never used 
a tool 


Less than 
once a 
week 


2 ~ 5 times 
per week 


More than 
5 times per 
week 


Total 


Blog 


79 


28 


2 


1 


110 


Wiki 


138 


18 


2 


2 


160 


Instant messengers 


7 


25 


25 


31 


88 


Online social networking (e.g., 


7 


8 


13 


72 


100 


Facebook) 












Online video sharing (e.g., YouTube) 


39 


39 


21 


9 


108 


Online games 


126 


30 


6 


7 


169 


Immersive virtual environment (e.g., 


239 


3 


0 


2 


244 


Second Life) 












Table 6. Analysis of usage time of Web 2.0 applications per week 


Web 2.0 applications 




Less than 


30-60 min 


More than 


Total 






30 min 




60 min 




Blogs 




108 


2 


0 


110 


Wiki 




155 


2 


3 


160 


Instant messengers 




45 


29 


14 


88 


Online social networking (e.g., Facebook) 




42 


28 


30 


100 


Online video sharing (e.g., YouTube) 




88 


15 


5 


108 


Online games 




153 


6 


10 


169 


Immersive virtual environments (e.g., Second Life) 


242 


2 


0 


244 



Table 7 shows the utilization level of Web 2.0 for learning on all seven Web 2.0 applications. On average, 
participants had a higher level of performance expectancy towards online video tools. Participants responded that 
using instant messengers, social networking, and online video sharing was very easy for them because they felt it 
unnecessary to put in much effort to use them. The anxiety level associated with using instant messenger and social 
networking tool are relative low on a 9-point Likert scale. However, participants felt intimidated by Social Virtual 
Environments. 

Table 7. The utilization level of Web 2.0 tools (9-point Likert scale) 
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Usage frequencies 
of Web 2.0 

Technology acceptance 


Blog 


Wiki 


Instant 

messengers 


Social 

networking 


Online 

video 

sharing 


Online 

games 


Immersive 

virtual 

communities 


Performance 


4.30 


4.79 


4.43 


4.93 


5.08 


4.09 


4.54 


Effort 


4.98 


4.88 


6.95 


7.18 


5.63 


5.09 


4.64 


Attitude 


4.63 


4.80 


5.43 


5.91 


5.85 


4.80 


4.64 


Social influence 


4.27 


4.65 


5.16 


5.40 


5.30 


4.53 


4.55 


Anxiety 


4.85 


4.84 


3.00 


3.02 


4.19 


4.36 


4.97 



See Table 8 for significant correlation analysis results between Usage frequency and Web 2.0 utilization 
levels. Use codes in Table 1 to identify corresponding items in the Web 2.0 utilization level survey. There is a 
positive correlation between usage frequency of Wiki and performance expectancy, effort, and attitude. Instant 
messenger, social networking tools, and online video sharing tools were very familiar to participants. There were 
several positive correlations between usage and performance expectancy, effort expectancy, attitude, and social 
influence of instant messenger, social networking, and online video sharing. In particular, there was a negative 
correlation between using online games and performance expectancy. In contrast, there were no correlations 
between the usage and acceptance of blogs and immersive virtual environments. 



Table 8. Correlations between usage frequencies of web 2.0 applications and technology acceptance (Pearson 
Correlation) 




*. Correlation is significant at the 0.05 level (2-tailed). 
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Table 9. Correlations between usage time of web 2.0 applications and technology acceptance (Pearson Correlation) 



Usage time of 
X^Web 2.0 

Technology \ 
acceptance \ 


Blog 
N=1 10 


Wiki 

N=160 


Instant 

messenge 

rs 

N=88 


Social 

networkin 

g 

N=100 


Online 

video 

sharing 

N=108 


Online 

games 

N=169 


Immersive 

virtual 

communiti 

es 

N=244 


Usage frequency 


- 


- 


.667** 


.523** 


.585* 

* 


.835 

** 


.926** 


Performance 


- 


.348* 

* 


.327* 

* 


- 


.253* 

* 


- 


- 


Effort 


- 


.315* 

* 


.357* 

* 


.428* 

* 


$ 

.481 

* 


.295 

** 


- 


Attitude 


- 


.342* 

* 


.390* 

* 


.206* 


$ 

.341 

* 


- 


- 


Social influence 


- 


- 


.462* 

* 


.434* 

* 


.265* 

* 


- 


- 


Anxiety 


_ 


_ 


_ 


_ 


_ 


-.234** 


_ 












.199* 







**. Correlation is significant at the 0.01 level (2-tailed). 
*. Correlation is significant at the 0.05 level (2-tailed). 



Table 9 shows the correlation between usage time and Web 2.0 applications. Performance expectancy 
affects the use of Wikis, instant messengers, and online video sharing tools. Students responded that they spent time 
using wikis, social networking, online video sharing and online game because they were easy to use. In addition, 
they spent time using instant messenger, social networking tools and online video sharing because they were 
influenced by their social environment. 



Discussion 

In terms of descriptive statistical analysis, it is encouraging to learn that most students found Web 2.0 
applications such as instant messengers, social networking, online video sharing, and online game easy to use. Most 
students were the least intimidated by several Web 2.0 applications in particular because they felt confident in using 
instant messenger, and social networking tool. Most students reported having a positive attitude towards using 
instant messenger, social networking, and online video sharing tools. However, blog, wiki, and immersive virtual 
environments were not familiar to participants. Most students reported the highest anxiety level in participating in 
blogs, wikis, and immersive virtual environment (eg., Second Life). 

Correlation analysis also generated numerous interesting finding. The results suggested that most students’ 
perceived performance expectancy on several Web 2.0 applications were correlated to the usage frequency. In the 
context of technology utilization motivation, this finding implied that extrinsic incentives (i.e., performance 
expectancy) seems to dominate the usage behaviors, which is congruent with the findings from other Web 2.0 
application studies, which indicated that Web 2.0 users are often driven by factors that are extraneous to the learning 
process. For example, Oulasvirta, Lehtonen, Kurvinen, & Raento (2009) suggested that motivation that drives blog 
users’ participation and contribution is mainly extrinsic (e.g., curious about others’ life, being acknowledged offline 
for efforts). Janzik and Herstatt (2008) identified that the combination of material and immaterial incentives (i.e., 
extrinsic motives) is likely to outweigh the intrinsic incentives. 

Although the findings suggested that students were very open towards using wiki, instant messenger, social 
networking tools, and online video sharing tools for learning, most students thought that using online games might 
affect their performance. The Digital Divide Model (Cooper, 2006, p.331) could explain this finding. The model 
postulates that the ongoing existence of gender difference when it comes to the utilization of computers and the 
Internet technology is the result of gender stereotypes. In this study since most of our participants were females, 



107 




their perceptions towards Web 2.0 applications might be influenced by such stereotypes that are so deeply rooted in 
their prior formal schooling experiences. Hence applications with more social support (e.g., social networking tools) 
are more popular than online games that are often masculine for our participants. 

Conclusion 

Given the small sample size it is inadequate to generalize our findings to different instructional settings. 
The preliminary results, however, offer potential research directions for us to continuously improve the study in the 
future. First, the selection process may be the direct result of students’ instructional preferences (e.g, collaborating 
learning activities, and individual learning activities) or the indirect consequences of available instructional 
resources and infrastructures. Future research needs to take both factors into consideration. Second, future studies 
should cultivate relationship between Web 2.0 application features and induced motivational support to create a 
customizable Web 2.0 learning environment design guideline in order to address different learning needs. Finally it 
is necessary to conduct in-depth task analysis for all seven Web 2.0 applications and align them with intended 
learning outcomes. As a result we will be able to optimize learners’ performance in Web 2.0-enriched learning 
environments. 
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Sustaining User Traffic in Open Course Ware Systems: A Case Study on the 
Open Knowledge Portal of Samsung Economic Research Institute 
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Abstract 

Open Educational Resource (OER) movement has reached its tipping point in recent years due to advancement of 
information and communication technologies (ICT). Among various projects, the OpenCourseWare (OCW) of MIT 
has inspired higher education institutions around the world to deploy OCW systems. The financial sustainability of 
OCW, however, has become questionable amid the economic recession. To adopt any revenue-generating models, 
the OCW must engage users to generate substantial user traffic. Hence, this case study, grounded in motivational 
theories of multimedia learning, analyzed the open knowledge portal of Samsung Economic Research Institute 
(seri.org), to identify features that motivate users’ continuous participations. The results include three 
recommendations to improve the design of OCW systems to better motivate and engage users. 

Keywords: Open Educational Resource, Open Course Ware 

Introduction 

Due to advancing information and communication technologies, opportunities for open learning have 
become pervasive via web-based open learning systems. This phenomenon exemplifies the Open Educational 
Resources (OER) movement in that quality educational resources become highly accessible to general public on the 
collaborative Internet. Among numerous projects that advance the OER movement, the OpenCourseWare initiative 
of MIT (2001) has started a trend in higher education institutions around the world, to make their course materials 
available on the Internet for everyone (Caswell, Henson, Jensen, & Wiley, 2008). Despite OCW’s popularity among 
general users and faculty and students in higher education in recent years, some, however, are concerned about the 
sustainability of open courseware systems. Issues such as the management of intellectual property, operational 
models of the technology, organizational and societal readiness for open knowledge sharing, and availability of 
infrastructure all play parts in sustaining OCW operations (Harley, 2008; Kumar, 2009; Schuwer & Mulder, 2009). 

A particularly pressing issue that could immediately impact the sustainability of OCW is its financial susceptibility. 
In the face of systemic economic downturn, institutions simply can no longer afford the operational cost of OCW 
(Harley, 2008; Parry, 2009). In reviewing a variety of financial models that OCW operations have adopted (Downes, 
2007) and current dire financial situations of higher education institutions, relying on external sponsorship to 
support OCW’s operational cost might be a relatively viable option in this economically challenging time. 

Similar to any other online content generation operations that depend on external sponsorships (e.g., 
Time.com, The Wall Street Journal Online), OCW needs to rely on user participations to generate revenues. That is, 
the user traffic of OCW systems must reach a large volume in order to attract external sponsorships (Rappa, 2010). 
Among reasons that prevent OCW from gaining a high level of user traffic, users’ lack of motivation to use the 
system was suggested to be an issue (Casserly & Smith, 2008; Harley, 2008; Simpson, 2008). To improve OCW’s 
ability to motivate and engage users in the online open learning environment, this conjectural case study intends to 
analyze Samsung Economic Research Institute (SERI )’ s open business knowledge portal (http://www.seri.org) with 
emphasis on how SERI.org sustains its user traffic via purposefully designed features that motivate and engage 
users. The objective of the case study is to inform OCW operations in higher education with feasible design 
strategies that could attract and retain a substantial level of user traffic thus enables the implementation of 
sustainable revenue-generating models. 

Literature Survey 

Open Higher Education 

The concept of open education, for individual learners, was emerged in the early 20 th century to advocate 
the openness of instructional climate thus promotes shared responsibility and freedom of self-expression among 
young learners (Peters & Britez, 2008, p.7; Walberg & Thomas, 1972). From the institutional viewpoint, open 
education consists of two levels of meaning. First, it represents the early era of distance education in higher 
education. For instance. The University of London External Study system opened its door to a wide range of people 
in 1836. What distinguished it from the Oxford University was its lack of residence requirement. Students of 
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University of London were able to conduct their academic assignments in remote locations and that was convenient 
for many UK citizens who were living on their colonies overseas. Today the Open University of UK continues to 
provide the same educational opportunities for people around the world (Tait, 2008). The second level of open 
education not only represents the “distance” factor of the teaching and learning process, but it also carries the 
mission of higher education beyond disciplinary and institutional boundaries. That is, open education should 
exemplify the philosophy of making educational materials readily available for the public good (Lane, 2008). With 
this level of open education, individuals can autonomously plan and execute their own educational processes from 
anywhere at anytime without any financial liabilities. This open learning approach further provides unlimited 
opportunities for informal and lifelong learning that benefit continuous professional development and global 
workforce development (Casserly & Smith, 2008; Huijser, Bedford, & Bull, 2008). Hence, many were inspired to 
share educational contents on the Internet and joined force with the Open Educational Resource movement. 

Open Educational Resource Movement 

The open educational resource (OER) movement was originated from the field of software development 
because some believe firmly that to facilitate the development of software applications, original programming codes 
should be shared with the community of developers. In the past decade, issues such as sharing mechanism, content 
licensing, and technological infrastructure in software development have been tackled with moderate success. Apple 
Computer’s Application Store (http://www.apple.com/iphone/apps-for-iphone/), for instance, is the result of sharing 
original programming codes with software developers around the world, which has generated thousands of 
application software for Apple products. 

In the meantime, the OER movement has expanded to numerous multidisciplinary collaborations for 
educational purposes. Practitioners and scholars from software engineering, information and communication 
technology, law, and policy studies have contributed greatly to advance the movement (Wiley & Gurrell, 2009). 
UNESCO (2002) captured the timely essence of OER by defining it as “open provision of educational resources 
enabled by information and communication technologies, for consultation, use, and adaptation by a community of 
users for non-commercial purposes.” The goal of OER movement is to level the access to knowledge via high 
quality and free online content for everyone in the world (UNESCO, 2009). Hylen (2005) further categorized OER 
into the following initiatives: 

• Open courseware and content 

• Open software tools 

• Open material for faculty of e-learning 

• Repositories of learning objects 

• Free educational courses 

Among these OER initiatives, open courseware has the closest tie with higher education institutions since it 
requires financial support from administrators and content sharing from the faculty. The OpenCourseWare (OCW) 
of Massachusetts Institute of Technology (MIT) is considered one of the pioneers who systematically share course 
materials online for free. 

The OCW of MIT 

Conceived in 1999, as part of the institutional effort to advance MIT’s educational technology capacity, the 
Institute decided to publish its course materials freely and openly on the Internet for everyone to use. This 
innovative initiative went online in 2002 and is now known as the OpenCourseWare (OCW)(Abelson, 2008). 
Currently the site hosts 1,900 courses and as of October 2009, the site has 86.8 million visits from users around the 
world (46% of overseas traffic). Among those users, 48% identified themselves as self-learners; 32% as educators; 
and 17% as students. The 2009 Program Evaluation of OCW (MIT, 2009) reported that at least 89% of its visitors 
were satisfied with the breath, depth, overall quality, and currency of the course materials. 

The launch and success of MIT OCW consequently encouraged other higher education institutions around 
the world to establish similar open knowledge portals. To maximize the efficiency of this inter-institutional 
collaboration, the OCW Consortium was created in 2005 and articulated the definition of OCW as, “a free and open 
digital publication of high quality educational materials, organized as courses.” (Carson, 2009). To date the OCW 
Consortium has 174 university members and 44 organization members from 37 countries (OCW Consortium, 2010). 

Challenges to Sustain OCW 

OCW has generated a significant level of interest in higher education as a practice to realize the 
knowledge-sharing mission of universities. Such innovative institutional initiative, however, often presents new 
challenges on many fronts. In the OCW Project of the United Nation University (UNU), for example, content 
organization within the portal emerged as an issue in the early stage of the project, which has direct impact on the 
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developmental cost. In terms of sources of contents, providing incentives for faculty members to participate turned 
out to be one major obstacle to overcome by the administration. During the development process, it was crucial for 
UNU to configure a standardized procedure to streamline the process while maintaining the quality of the course 
content (Barrett, Grover, Janowski, Lavieren, Schmidt, & Ojo, 2009). 

Financial concern is another major challenge for institutions to uphold OCW operations. In 2007, MIT 
reported a steady-state annual cost of $4M to operate its OCW portal with 50% of which relying on external 
funding. But it is uncertain if the external funding would be allocated to OCW operations in the future (Abelson, 
2008). Downes (2007) discussed several financial models for higher education institutions to sustain OCW projects. 
The majority of them, however, are only feasible when the economy is sound. For example, the Donation Model, 
Institutional Model, and Governmental Model, to a large extent, all depend on the goodwill of the society and 
communities to provide financial support. Unfortunately the funding disappears when the economy comes to a 
recession (Harley, 2008). The Sponsorship Model, on the other hand, could be a viable option for OCW projects to 
embrace in this challenging time for several reasons. First, this model has been proven effective to financially 
sustain similar open content operations via radio or television programming. Sponsors (or advertisers) would pay 
fees to post messages to promote new products or services during the program. Second, the source of funding is no 
longer associated with institutional and governmental resources that are continuously shrinking and are vulnerable to 
systemic economic risks (e.g., low employment rate). Instead, the financial support is directly from businesses and 
enterprises that utilize the Internet for marketing purposes. In 2009, the overall spending of online sponsorship in the 
U.S. was estimated as more than $25B with the annual growth rate of at least 10% till 2013 (Shankland, 2008). The 
Stanford on iTunes project is an example of the Sponsorship Model in practice, which connects open content users 
with the iTune application of Apple (Downes, 2007, p.35). Certain conditions, however, must be met in order for the 
Sponsorship Model to be successful. 

One critical condition for the Sponsorship Model to be viable is having substantial number of user visits 
within a limited time period (i.e., user traffic), which demonstrates the open knowledge portal’s impact level to 
potential sponsors (Rappa, 2010). In terms of factors affecting OCW usage and financial sustainability, it was 
suggested that the lack of understanding on OCW users might increase the operational cost of OCW systems. Hence 
it is crucial to know who the users are, why they use OCW, and how they are using OCW (Harley, 2008). Casserly 
& Smith (2008) further suggested that OCW users’ intention in contributing and sharing the content might play a 
role in sustaining OCW operations. In terms of users motivation, Simpson (2008) argued that it is important to 
provide proactive motivational support to users of open learning systems since they might be less engaged in the 
learning processes. Therefore to motivate individual users in OCW, from the viewpoint of instructional system 
design, the interaction between OCW content and users in the open learning environments should be purposefully 
designed just as designing any other online learning environments. OCW’s motivational support to engage users and 
its potential effect on users’ cognitive load both need to be taken into design considerations. 

Motivational Support and Cognitive Load of Using OCW 

The design of OCW should consider users’ motivational level in order to effectively engage them. Situated 
in a workplace setting, Davis, Bagozzi, & Warshaw (1992) validated the relationship between intrinsic motivation, 
extrinsic motivation, and the usage of computer-based technology. In their study the perceived usefulness 
contributes to users’ extrinsic motives and the enjoyment of usage accounts for the intrinsic motives. ChanLin 
(2009) applied the ARCS model for motivational design (Keller, 1987) in designing a web-based course and 
concluded that motivational design in online environments is as effective and necessary as applied in face-to-face 
settings. Astleitner & Wiesner (2004) asserted that only considering intrinsic motivational factors of Malone and 
Lepper (1987)(e.g., challenge, curiosity, rules, fantasy) is insufficient to explain the motivational outcome of the 
interactive and self-regulated learning process in multimedia computer-based learning. Motivational processing that 
illustrates the dynamic relationship between users’ perceived motivation and the intended content information must 
also be included in the design. Furthermore, users’ cognitive load also needs to be included when designing the 
interaction between users and the open content, to ensure that users do not get overwhelmed by the demand of 
cognitive processing (Ang, Zaphiris, & Mahmood, 2007; Sweller, van Merrienboer, & Paas, 1998; Warschauer, 
2007), which could deter users’ future visits. Therefore the design of OCW should engage users cognitively via a 
variety of strategies. To achieve the abovementioned design objectives, the theory of Motivation, Volition, and 
Performance could be utilized to guide the design process in that it integrates intrinsic motivation and cognitive 
information processing to explain the iterative process of motivation development in relation to learners’ final 
learning outcome and performance (Keller, 2008). 
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Integrative theory of Motivation, Volition, and Performance 

The integrative theory of motivation, volition, and performance (MVP) argues that a complete motivational 
learning cycle is consisted of several stages to encompass learners’ motivational, cognitive, and outcome processing 
(Keller, 2008, p.94). The motivational processing helps learners set up initial performance goals that are critical for 
sustainable learning processes. Learners at first should have sufficient level of curiosity to explore the learning task 
(attention); then understand the value of the learning task (relevance), and evaluate the possibility of attaining 
successful performance (confidence), to identify and confirm the performance goal. These processes, in turn, 
prepare learners for the follow-up actions of learning. 

Another aspect of MVP theory focuses on learners’ cognitive information processing. At the effect of 
motivational processing learners actively apply meta-cognitive strategies to carry out learning activities to help them 
create and automate transferrable mental models. The cognitive processing capacity, however, is limited by learners’ 
working memory. The result of cognitive processing hence feeds into the outcome processing stage, which allows 
learners to evaluate the discrepancy between the performance consequence and their invested efforts. Learners 
reflect upon all previous stages’ experiences emotionally and cognitively, and develop a collective sense of 
satisfaction towards the learning process. 

In the context of designing motivational and engaging OCW systems, the implication of the MVP theory is 
threefold. First, since motivational processing is crucial at the early stage of the learning process, the design must be 
cautious to neither overwhelm users’ processing capacity nor distract them with competing stimuli. Second, users’ 
cognitive processing activities could play a substantial role in sustaining their iterative motivation. Users overloaded 
with cognitive stimuli, regardless of their initial motivational processing results, are still vulnerable to be 
unmotivated by exhausting cognitive information processing tasks. Third, the result of outcome processing could 
impact the engagement of OCW systems. Unsatisfied users usually do not come back to use the same system since 
they do not perceive the process worthwhile, which in turn, will decrease the overall user traffic to the OCW site. 
Purpose of the Study 

To inform the design of OCW systems, this study employs conjectural analysis to align features of existing 
open knowledge portals with design strategies suggested by the theory of MVP. Prior studies have utilized similar 
analysis approach to develop feasible design strategies for complex multimedia learning environments (Dickey, 
2007; Kebritchi & Hirumi, 2008). The present study, applying the same conjectural analysis approach, discusses 
features of Samsung Economic Research Institute’s (SERI) online knowledge portal in relation to the MVP theory. 
The purpose of this study is to identify applicable design strategies in open knowledge portals to motivate and 
engage users. It is our hope that by incorporating motivation-inducing design strategies, OCW systems are able to 
generate a substantial volume of user traffic, hence, will make certain financial models viable to sustain the OCW 
operation. 

The Case Study 

Samsung Economics Research Institute (SERI) 

Samsung Economic Research Institute (SERI) was established in 1986 by Samsung Group, the largest 
conglomerate in South Korea, to serve as a think tank for the country’s economic policy development. Its mission is 
to make the society a knowledge powerhouse by creating a dynamic mass of knowledge objects (SERI World, 

2009) . Currently the Institute generates $100M of revenue per year by providing human resource management and 
business strategy consultations and knowledge generation services to its paying clients. The characteristics of 
SERI’s knowledge objects are described as timely, easy to comprehend, relevant to societal issues, user-centered, 
highly interactive with general public, and influential to governmental policy development. The critiques of SERI, 
however, argued that its knowledge objects tend to focus on non-serious issues and the content does not provoke in- 
depth debates and discussions (Chung, 2009) 

SERI’s Open Knowledge Portal 

SERI launched its open knowledge portal (http://www.SERI.org) in 1996 in Korean and it has been ranked 
as the top “knowledge ecosystem” in South Korea. SERI.org currently has 1.68M of registered members and hosts 
2.5M knowledge objects in texts, narrated presentations, video clips, and user-contributed content. The membership 
is open to everyone for free. Premium content is also available for a fee. Among these knowledge objects, 10% of 
them are produced by SERI staff; while the rest of them are contributed by SERI.org members in derivative forms. 
The content covers subjects from business strategies to daily problem-solving job aids for working adults (SERI, 

2010 ) . 
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SERI World: The Asian Corridor 

In recognizing Samsung Group’s close business relationship with its international partners, SERI launched 
SERI World in 2004 to serve a wider audience around the world. The content on SERI World has the same origin of 
SERI.org but it is all written in English. In early 2009, after five years of operation, SERI World has acquired 
55,000 members worldwide including nearly 20% who are non-Korean nationals (Han, 2009). SERI World covers 
international economics, management, industries, and political issues in various media formats (HTML pages, PDF, 
video reports). 



Design of SERI.ORG 

Upon reviewing the design ofSERI.org, we have identified several features that are absent in existing 
OCW systems (e.g. MIT, Yale University, John Hopkins University, Kyoto University). The following section will 
describe each feature in detail. 

Open membership. Users can register for the SERI.org membership for free. The process is as simple as if 
you are setting up a social media account (e.g., the Facebook). See Figure 1 for the screenshot of SERI.org. Figure 2 
shows the interface for SERI World. Members’ account management page is shown in Figure 3. The interface 
design ofSERI.org, as you can see from the screenshot, is fairly easy to navigate, as the content topics are not all 
cluttered in one column. The page also uses vivid color contrast arrangement to help users better locate the links. 
The graphic on the home page might also trigger users’ curiosity and intentions to explore further. On the SERI 
World home page, users also can easily locate the Top 5 accessed knowledge objects on the top-right corner of the 
page. On the member’s account page, the features enable users to self-monitor and manage their learning activities 
in SERI.org, which is essential to support users’ intrinsic motivation (Malone & Lepper, 1987). 
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Supporting cognitively efficient learning processes. When delivering content, SERI.org repackages 
original research reports into a variety of formats. One unique approach that SERI.org takes is to condense lengthy 
reports into a short multimedia video clip that is directly streamed from SERI.org to user computers. This delivery 
method can effectively manage the cognitive load imposed on users since the video clip is purposefully designed to 
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have approximately five minutes in length. Furthermore, it enables users to access otherwise arcane and dull 
research reports by presenting the knowledge in the context that is convenient and relevant to users. If users are 
interested in learning more about the topic presented in the video clip, they can further retrieve the original research 
studies on SERI.org. Finally, users get to rate the overall quality of the video knowledge object using a rating 
system, which encourages users to reflect on the video viewing experiences. Figure 4 shows the five-minute video 
object example with rating system. 
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Web 2.0 features to engage users. SERI.org takes the advantage of Web 2.0 technologies to engage users. 
Users can follow certain bloggers within the SERI.org community. They also can leave comments in response to 
blog postings. This feature exemplifies the concept of Web 2.0 in that every user can contribute to the content 
collectively and new ideas might be derived from such collaborations (Anderson, 2007; O’Reilly, 2005). Figure 5 is 
the blog page. See Figure 6 for the user commenting function in SERI World. 

In addition, similar to applications of social bookmarking systems (e.g. delicious.com), users ofSERI.org 
can categorize the content of blogs with tags. The tagging process may facilitate learning because (1) it prompts 
users to reflectively summarize ideas based on his or her existing knowledge base, (2) it enhances peer interaction 
by allowing users to compare his or her tags with peer users, and (3) it helps users to locate relevant content by 
clicking on the tags (Fu & Kannampallil, in press). This feature not only enables users to retrieve content more 
efficiently, but also it allows users to share different perspectives on the same knowledge object. From a cognitive 
learning perspective, the tagging function could reduce the demand of cognitive processing for users as they can 
easily connect to peer users’ mental models via tags (Siemens, 2005). 
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Incentivizing user participation. SERI.org uses two strategies to encourage user participations. First, 
users receive Flower Points for their participations in ongoing online postings or discussions. Different flower 
points are awarded to users based on different types of participations in contributions to different knowledge objects. 
There are two categories for users to receive flower points. In the first category users automatically receive the 
appreciative flower points by simply participating in the online forums. In the second category users can receive 
additional flower points based on the quality of their contributions to the online spaces. The SERI staff is 
responsible of evaluating and awarding those value-based flower points. Table 1 shows different levels of flower 
point awards according to contributions to different knowledge objects and activities. Once users accumulate enough 
flower points, they can use the points to unlock otherwise unavailable knowledge objects. For example, users can 
“purchase” books published by SERI with at least 200 flower points. (See Table 2) Figure 7 shows the flower point 
on one user’s account. Moreover, SERI.org recently launched another feature to show the ranking of members’ 
overall flower points. The interface was designed to mimic the trading chart of stock markets. (See Figure 8) The 
second strategy is that SERI.org hosts the Most Frequent Knowledge User contest every month to recognize users’ 
participations. The winners are introduced to all SERI.org members. See Figure 9 for a previous winner of the 
contest. 
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Table 1 

Flower Point Awards Based on Different Levels of Contribution 





Flower Points to Appreciate Participation 


Flower Points for the Value of 
Contribution ** 


Where 

Menu 


Open Knowledge Zone 


Cyber Forum 


Open 

Knowledge 

Zone 


Cyber 

Forum 


Research 

Proposal 


Posting (100) 


Posting 

*M (50)+, F (100) 


(1-500) 


M (1-500)+ 
F (1-500) 


Issue Debate 


Posting (50) 


Posting 

M(30)+,F (50) 


(1-300) 


M (1-300)+ 
F (1-300) 


Open Research Lab 


Posting (100) 


Posting 

M (100)+, F (200) 


(1-300) 


M (1-300)+ 
F (1-300) 


Cyber Research 


Applying for starting a 


Applying for starting 


(1-500) 


M (1-500)+ 


Group 


group (100) 

Opening the group 
(200) 

Participating as a 
researcher of the group 
(30) 


a group 

M (100)+, F (200) 

Opening the group 
M (50)+, F (100) 

Participating as a 
researcher of the 
group 

M (30)+, F (50) 




F (1-500) 


Knowledge 


Posting 


Posting 


(1-500) 


M (1-500)+ 


Expert 


(10) 


M (10)+, F (20) 




F (1-500) 


Benchmarking 


Posting 

(100) 


Posting 
F (200) 


(1-500) 


- 


Survey Center 


Responding to a 
research survey 
(50) 

Answering to an online 

poll 

(5) 








Best Forum 
Contents 


- 


- 


- 


M (50)+, F 
(100) 


Updating personal 
information 
Log-in to Seri.org 
(daily) 


(50) 


(100) 






Creative idea 


Posting (100) 


- 


(300) 


- 


Business Model 
Proposal 


Posting (100) 


■ 


(300) 


“ 


Acrostic Poem 


Writing a poem (30) 


- 


(300) 


- 


Mensa Quiz 


Trying (100) 
Solving (300) 


- 


- 


- 



*M: Member, F: Fonim 

**The SERI staff evaluates the quality of the contribution and then rewards flower point. 
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Table 2 

Flower Points Needed for Advanced Knowledge Objects 



Knowledge Objects 


Flower Points Needed 


Video clips 


50 (Users have 24 hrs to view the clip.) 


Books published by SERI 


200 ~ 400 per book 


SERI CEO annual membership (excluding 
offline service such as breakfast meetings) 


1,000,000 


SERI Spark annual membership (a paid 
website for young employees) 


300,000 
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Figure 8. The Flower Points ranking 
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Discussion 

What distinguish SERI.org from other open knowledge portals are its abilities to engage and motivate users 
to continuously visit the site, which would be the key element for OCW to sustain its operations with the 
Sponsorship Model. The following sections will discuss features ofSERI.org in the context of learner motivation, 
efficient cognitive learning, and the MVP theory. 

Motivation to Learn, Participate, and Contribute 

Goal-directed behaviors are often stimulated and maintained by the process known as motivation (Schunk, 
1990), which is the essential element to initiate and sustain learning and performance. Multimedia computer-based 
learning environments therefore need to be designed with care to provide adequate level of motivational support 
(Ames, 1992; Anderman & Maehr, 1994; Bandura, 1997; Berliner & Gage, 1998; ChanLin, 2009; Sachs, 2001; 
Sankaran & Bui, 2001; Weiner, 1985). More importantly, one must consider features that not only attract users in 
the beginning, but also engage users to keep using the system for as often as they possibly can. That is, the design of 
the system should prompt users’ frequent visits. Users’ motivation to use the system needs to go beyond one single 
visit. The design must be able to sustain users’ motivation between visits. This issue is particularly critical in OCW 
systems because the time gap between users’ visits could be substantial. In the theory of MVP, that is why the result 
of outcome processing is so critical since it helps learners determine if the learning task is worth the effort thus leads 
to the continuation or termination of the follow-up learning tasks. A favorable outcome processing result would 
motivate users to continue on with the learning process using the same OCW system. To sustain this self-regulated 
learning processes in multimedia learning environments, the design of learning environments must consider both 
learners’ intrinsic and extrinsic motivation (Astleitner, & Wiesner, 2004; Davis, et ah, 1992). 

In terms of intrinsic motivation, to achieve the optimal level, the design of learning environments should 
utilize elements of challenge, control, rules, and fantasy (Malone & Lepper, 1987). The design ofSERI.org might 
have appropriate support for developing intrinsic motivation as it always adds new content to challenge users, allows 
users to pace themselves in going through self-selected knowledge objects, and provides clear rules in participating 
in the open knowledge portal. The element of fantasy, however, is lacking in its current design. 

With regards to extrinsic motivation, the design ofSERI.org has outperformed existing similar operations 
(e.g., OCW of MIT). In particular the Flower Point system ofSERI.org has played a significant role in enhancing 
extrinsic motivation. Users ofSERI.org not only receive flower points from all their activities on the site, but also 
they can exchange the flower points with tangible rewards (e.g., books by SERI). In the context of learning, extrinsic 
motivation drives learners’ behaviors to attain rewards or avoid punishments that are external to learning processes 
(Keller, 1983). Therefore the flower points of SERI.org, to a large extent, might initially encourage user 
participations in different activities. More importantly, the flower points engage users with external rewards thus 
make them keep coming back for more. In the context of MVP theory, the Flower Point system feature might induce 
positive results of the outcome processing by providing tangible extrinsic rewards that are attached to all users’ 
participation and contribution activities. Hence, users are more likely to revisit SERI.org for their future self- 
learning tasks. The product of such engagement could be easily translated into traffic-driven financial models, to 
sustain OCW operations in higher education institution. 

Cognitively Efficient Design of Knowledge Objects 

As suggested by Cobb (1997), a cognitively efficient process needs to utilize multimedia to help learners 
process extraneous information. This position was further supported by principles of multimedia learning proposed 
by Mayer (2001). The short video clips ofSERI.org exemplify these design principles for cognitively efficient 
learning processes because they are able to deliver otherwise complex and dull content with multimedia 
presentations that are easy for users to process and apply. With approximately five minutes per video clip, users are 
able to explore numerous topics within limited amount of time without overloading their cognitive processing 
capacity. 

The other feature that supports the efficient cognitive learning process is the tagging function of SERI.org 
blogs. The blog content is usually stemmed from SERI’s research publications, which puts the research finding in a 
context that readers can relate to. By allowing the tagging function on the blogs, not only can users search for the 
content easily, but also they can see how others might interpret the content with different tags thus promoting 
reflective learning (Fu & Kannampallil, in press). 

In the context of MVP theory, since cognitive processing is between learners’ motivational processing and 
outcome processing, its role in sustaining desired learning processes is crucial. A taxing cognitive processing could 
discount the positive result of motivational processing thus leads to unfavorable outcome processing results. The 
short video clips and tagging features could substantially reduce the cognitive load of learning in SERI.org. 
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Accordingly learners are less likely to feel overwhelmed by conglomerates of knowledge objects available on the 
open portal, hence enables efficient and sustainable learning processes. 

Conclusions and Recommendations 

In light of user motivation’s role in impacting OCW systems’ financial sustainability, this study was set out 
to identify feasible design strategies that could promote user motivation in similar open knowledge portal settings. 
Grounded in the theory of MVP, the case study on SERI.org has reached three recommendations for a sustainable 
OCW system design. First, the design must utilize the power of extrinsic rewards to engage users. The OCW system 
must cultivate users’ responses to extrinsic incentives to sustain their continuous usage of the same OCW system. 
The Flower Points system of SERI.org is a great example of using extrinsic rewards to engage users. Second, the 
design of OCW should include Web 2.0 technologies to support the interactive social learning process. Tools such 
as blogs, wikis, social networking tools, and user-moderating forums can easily provide opportunities for users to 
initiate and participate in organic social collaborations. Web 2.0 technologies further enable OCW users to create 
communities of practice based on common interests thus sustain the network of informal and lifelong learning 
(Carson, 2009). Finally, with the multimedia capacities of the Internet-based applications, the design must consider 
learners’ limited cognitive information processing capacities. That is, the learning process induced by the open 
content should not overwhelm learners cognitive load. That is, the content should be easy to search, locate, retrieve, 
and retain. Furthermore, multimedia elements and social interactions need to be implemented based on proven 
design principles in multimedia learning environments. 
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Abstract. 



E-books are gaining popularity for personal reading. Options for access to large selection of book titles and 
anytime/anywhere reading choices add to the increased use of e-books. For this study, students received satisfaction 
surveys on four separate reading sessions— one print-based and three e-book titles. Indicators of reading engagement 
include motivation for independent reading, dialog with other students about reading materials, fluency, and 
comprehension as measured by a standardized tests on the print book and all three e-books. Results showed format 
is not as important as students’ identification with setting, characters, and theme of the book, however, students 
indicated a preference for e-books when given the option of a wide selection of titles and the freedom to choose their 
own e-book. 

A number of characteristics factor into the broader construct of reading engagement in elementary-aged 
children. Reading engagement is comprised of factors such as reading motivation (Clarke, Power, Eloffman, 

Kelleher, & Novak; 2003), home environment (Arzubiaga, Rueda, & Monzo, 2002), independent reading, and 
reading achievement (Wigfield, Guthrie, Perencevich, Taboada, Klauda, Mcrae, and Barbosa; 2008). Reading 
engagement is multidimensional and is influenced by a number of factors including cognitive and emotional 
engagement. Children who use sophisticated strategies and enjoy literacy activities are considered to be engaged 
readers (Wigfield, et. al., 2008). Consistently engaged readers actively seek appropriate books and become excited 
about learning new material (Lutz, Guthrie, & Davis, 2006). Marinak and Gambrell (2008) also found that children 
are motivated to read and remain engaged in reading when rewarded with the opportunity to choose their own 
books. Arzubiaga, et. ah, (2002) found context within literacy activities to be an important factor and is crucial to 
reading engagement and literacy development. They discovered that the culture of the school, classroom, 
intervention programs, and home will all influence a child’s reading achievement. The reading material that is 
available in the home and in the classroom will have a determining role in children’s long-term reading engagement. 
The materials that parents decide to keep in their home (Arzubiaga, et. ah, 2002) or the material that teachers elect 
for the classroom (Flowerday, Shaw, & Stevens; 2004) are crucial in shaping the literacy development of children. 
Children are more likely to become engaged in reading if they have greater access to books through home, school, or 
public libraries and are able to witness engaged reading by adults (Jewell, Phelps, & Kuhnen, 1998). Teachers can 
support children’s reading engagement if they provide ample opportunities for independent reading. Opportunities 
for independent reading will build fluency and allow children to increase their level of confidence (Kasten & 
Wilfong, 2007). Support for independent reading is crucial for the reading success of children with limited English 
proficiency (McGlinn & Parrish, 2002). Flowerday, et. ah, (2004) also found that children’s reading engagement is 
positively affected if there is high-interest material available for children even when children are not afforded the 
opportunity to self select their reading material. 
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All of these findings indicate that the material available to children will play a determining factor in how 
engaged children are in reading. With the influx of computers and other forms of technologies in the classroom in 
conjunction with programs such as Accelerated Reader™, the classroom culture is becoming enhanced through the 
use of digital resources (McGlinn & Parrish, 2002). Children are now being acclimated to technology in their 
classrooms beginning with preschool and achieving through adulthood. These technologies facilitate deep learning 
and critical thinking skills in children. Critical thinking with the use of the new technologies supports constructive 
peer collaboration and self-assessment in each learner (Hyun, 2004). This new classroom culture will ultimately play 
a role in children’s literacy development. Therefore, computers and other technology are likely to shape the way that 
children will ultimately view literacy (Mioduser, Tur-Kaspa, & Leitner, 2000). 

This is all taking place in the midst of a major shift in the publishing industry as many book sellers are 
beginning to actively promote e-books and e-book readers as a less expensive and more efficient method to read. 
Amazon.com introduced the Kindle in November of 2007. This devise allows readers to download books, 
magazines, and newspapers in less than one minute (Binder, 2008). Amazon has more than two hundred thousand 
titles available for download. The cost of the Kindle continues to drop (Sangani, 2009). Many current bestsellers are 
available for at significantly reduced cost (Binder, 2008) and more than a million out-of-copyright books available 
completely free of charge (Lardinois, 2009). Not to be outdone, Apple and Sony have both introduced e-readers to 
compete with the Kindle. Apple offers the iPod, and Sony offers the PRS300 at a lower cost than the Kindle 
(Bairstow, 2009). Like the Kindle, these devices will enable readers to download and read PDF files and have Broad 
Band access (Binder, 2008). 

Electronic books are available through two main sources; online websites and personal electronic devices. 
The importance of such devices cannot be underestimated. There are several factors influencing the transition from 
print to electronic format. First, authors are beginning to realize that electronic publishing gives them complete 
control over their artistic creations and will allow publication of their own material in the absence of an established 
book publisher by making books available on personal websites and charging patrons a fee to download the books to 
their e-reader devices. This enables them to retain 1 00% of the profits instead of the ten to fifteen percent that they 
usually earn by publishing through a book publisher (Sangani, 2009). There are even rumblings about publishing 
sites comparable to social networking sites such as YouTube.com where writers can share their books with a 
community of writers (Sangani, 2009). The increasing number of e-reading devices and the authors’ embracing of e- 
books would indicate that the technology is more than a passing fad. Economic downturns in our society and 
increased use of personal computing devices have caused some lessening of the tight control of many major 
publishers (Kaufman, 2010). 

E-readers have yet to forge their way into America’s classrooms; however children are being exposed to 
electronic media in a variety of ways during the school day. A growing number of online resources are available for 
classroom use (Brown & Hill, 2009). Currently the selection of titles is somewhat limited however there are many 
benefits in the use of e-books available through well known publishers. One such example is Scholastic, Inc. Access 
through subscription makes it possible for elementary children to view popular fiction books presented through 
dynamic multimedia websites. Each fiction book is aligned with a nonfiction content area book that provides that 
provides meaningful context for the animated stories. These books are available online at Scholastic.com. 
Additionally, there are a number of free websites that provide well-known fiction and picture books for viewing and 
reading. For example, the Screen Actors Guild Foundation has developed a website with well-known movie and 
television artist. These “read-aloud” sites make it possible for celebrity role models to read to any child with access 
to a computer and an internet connection (Available online http://www.storylineonline.net/). 

The combination of new-found popularity for electronic reading and computers in classrooms would 
indicate that these factors will shape children’s literacy development in the next decade. Children are still being 
introduced to literacy through print books, but signs point to electronic reading being a greater part of their literary 
life as they develop and mature. If the goal of schools is to keep children actively engaged in reading, then educators 
need to examine the effects of electronic books on the reading engagement of children. As has been stated, 
engagement also impacts motivation and reading achievement (Marinak & Gambrell, 2008). If the publishing trend 
of promoting electronic books continues, then it is reasonable to predict these books will eventually make their way 
into the schools. The purpose of the present study was to examine the reading engagement and comprehension of 
children as they read electronic books. 
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Research Questions. 

The primary research question was “What are the effects of electronic books on third-grade children’s 
reading engagement?” Because engagement impacts comprehension and the degree of satisfaction that children gain 
from reading, two sub questions were examined. First, “What are the effects of electronic books on third-grade 
children’s reading comprehension?” and second, “What are the effects of electronic books on third-grade children’s 
reading enjoyment?” 

Significance. 

The study has the potential to determine if the features of electronic books will actively engage children in 
reading material. Many electronic books are accompanied with features such as audio of entire text or audio of 
specific vocabulary terms. Definitions of such terms are also provided. The books are therefore interactive and allow 
for children to become actively engaged in the text. By introducing children to electronic books at an early age they 
will gain experience with all of the characteristics of such technology and become acclimated to twenty-first century 
devices for reading. Children will also gain valuable twenty-first century skills in the area of information and 
communication technology (ICT). The ability to use digital resources is a major factor in the success of the future 
workforce. 

Methodology. 

The participants were 22 third-grade children in a single self-contained classroom at an urban school in the 
southeast region of the United States. The students consisted of 1 1 boys and 1 1 girls. Four of the students were 
Caucasian, one student was Asian-American, and all of the remaining students were African-American. Most of the 
students read on grade level, although a few read below grade level. The students were divided into four groups for 
the study with a mixture of boys and girls in each group. The students were also grouped homogeneously by reading 
level. The children were ultimately rewarded with a free pencil and cupcakes for their participation in the project. 

The Procedure. 

Permission was obtained from the parents of each participant. The teacher, school principal, school 
division, and university Institutional Review Board also all granted permission for the study. All supplies were 
provided by the two researchers with the exception of the laptops which belonged to the school. The participants 
were introduced to the project, and verbal assent was obtained from the students. 

Phase I: The project began with the students reading a traditional print version of The Yellow Flouse 
Mystery (The Boxcar Children, No. 3) by Gertrude Chandler Warner and Mary Gehr. The children read aloud in 
their respective groups with either a researcher or one of two undergraduate research assistants to monitor their 
reading. The participants read the first three chapters of the book using a process called “bump reading” in which a 
child reads aloud as long as she or he wants and then calls on another child in the group to read. The process 
continues until the passage has been completed. 

Wigfield, et. ah, (2008) found that reading engagement was a construct that could be measured primarily 
based on the combination of reading achievement and reading motivation. Those researchers developed the Reading 
Engagement Index (REI) to measure engagement in elementary-aged children. Using the criteria described on the 
REI the researchers developed an observation instrument to be used during read-aloud sessions for both print books 
and ebooks used in the study. Researchers videotaped the sessions and made note of particular reading skills. The 
noted skills, as identified on the REI were as follows: 1) Works hard in reading; 2) Engaged during the lesson; 3) 
Works at decoding skills; 4) Is a confident reader; 5) Decoding skills applied; and 6) Fluency. A running tally was 
recorded each time a participant displayed one of the skills while reading aloud. 

Upon the completion of each of the first two chapters of The Yellow House Mystery, the student groups 
were given a reading activity by the classroom teacher to measure their comprehension and predicting skills. The 
students worked in their respective groups to complete mapping activities and presented their maps to the class. 
Students then read chapter three silently. After reading chapter three, students were administered a comprehension 
test on chapters one through three and an enjoyment survey of the first three chapters. Students were asked to enter 
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their name on the survey so information could be cross tabulated with their comprehension score and observation 
data. Since students may have concerns about impact on reading grades, the researcher assured students that the 
scores on the comprehension test or survey would not affect their school grade in any way. 

During Phase II of the study, each student was provided with a school-owned laptop. Participants were then 
introduced to the website Raz-Kids.com and provided instructions for the login process. Each participant was given 
login identification and a password. Students were also informed that they would be able to log into the site and read 
books at home on their own time. A researcher provided instruction in interactive features of the books on the 
website. Students were then given opportunities to investigate and experiment with the online features. These 
included electronic page turning, vivid color, graphics, and access to a library of approximately 100 titles grouped 
by reading level. Each title had two options for viewing. The students could listen to a computerized read-aloud 
version or view text and pages as ebooks with links to glossary and selected words that include audio 
pronunciations. The design of the study required students to access only the assigned books while data was 
collected, however, it was later discovered that access to a library title was a strong motivator for engagement. 

After instruction on the website and its features, students were then instructed to rejoin their original groups 
and to bump read a story called The Mystery Wind by Cheryl Ryan and Hough Armstrong from the website. One 
student read at a time, while others followed along. The student who read aloud at any given time was the only one 
with permission to use the interactive features at the time. Others in the group were instructed to continue reading 
along silently until it was their turn to read. Again, the observers videotaped the sessions and made notes on the 
aforementioned reading behaviors. After completing The Mystery Wind students were again assigned a 
comprehension activity by the classroom teacher. Students completed the activity in their respective groups and 
shared their ideas with the class. Students then completed a comprehension test in the same fashion as the test in 
Phase I. They also completed a survey to measure their enjoyment of The Mystery Wind. 

During Phase III of the study, students silently read another book from Raz-Kids.com entitled The Sweet 
Potato Challenge by Vera Ogden Bakker and Joel Snyder. Students were allowed to use any of the interactive 
features while reading the book. Upon completion, students were then given a comprehension test and administered 
an enjoyment survey of the book. Students were also administered a fourth survey to gage their overall enjoyment of 
reading electronic books. Students were finally given an opportunity to read or listen to any book from the website 
and play with all of the features as they read. All materials were then collected by the researchers and undergraduate 
students. Students were informed that they would be able to read books from Raz-Kids.com from home using the 
same login I.D. and password. 

Data Analysis. 

The primary data were the scores on the comprehension tests and the answers to the survey items. The 
observation data were secondary. All comprehension tests were created by the researchers based on research by 
Johns and Lenski (2005). These authors concluded that effective assessment of reading comprehension hinges upon 
students being able to a) preview the text, b) activate prior knowledge, c) identify main ideas, d) sequence, e) make 
predictions, f) make inferences, and g) draw conclusions. At least one question was included in the comprehension 
tests to assess each one of the aforementioned cognitive skills. The researchers also scored each assessment and 
entered the data into a statistical processing program. The initial three surveys were designed to measure the a) 
students’ level of enjoyment with each selection, b) the ease with which they read, c) their self assessment of 
comprehension, d) their motivation to read more of each selection, e) their desire to read other comparable books, f) 
the likelihood of reading the book outside of school, g) the recommendation they would give to a friend about the 
selection, and h) their satisfaction with the selection. The final survey was designed to measure their preference for 
traditional print books or e-books. The survey items were assigned a value and entered into the same software 
program. A repeated measure ANOVA was performed on the comprehension test data to measure the variation in 
test scores depending upon the format of the book. The survey data were cross tabulated with the comprehension 
data to measure the interaction between their enjoyment of the books and their comprehension scores. Ch-Square 
was used to identify any relationships between preferences for selected titles and format of books. 
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Results. 



The repeated measure ANOVA revealed no significant difference in the mean scores between 
comprehension tests one and three. However the mean score on comprehension test two differed significantly from 
the other two tests. Table 1 provides mean scores for the three tests. 

Table 1 . Comparison of mean scores for the three reading comprehension tests. 



Estimates 


Measure: MEASURE l 


Test 


Mean 


Std. Error 


95% Confidence Interval 


Lower Bound 


Upper Bound 


1 


80.262 


3.805 


72.348 


88.175 


2 


65.800 


3.735 


58.032 


73.568 


3 


81.625 


3.990 


73.328 


89.922 



The post hoc test was used to conduct pairwise comparisons. The pairwise comparisons revealed levels of 
significance among the three mean scores. The difference between test one and test two yielded a significance value 
of 0.023, p > .05. The mean score on test two also yielded a significant difference from test three with a significance 
value of 0.002, p > .05. The mean scores from tests one and three were not significantly different. The level of 
significance was 1.00,/? > .05. 

Table 2, Pairwise tests providing level of significance in mean score differences. 

Pairwise Comparisons 



Measure : MEASURE 1 



(I) Test 


(J) Test 


Mean Difference 

(I-J) 


Std. Error 


Sig. a 


95% Confidence Interval for 
Difference 3 


Lower Bound 


Upper Bound 


1 


2 


14.462* 


4.889 


.023 


1.744 


27.180 


3 


-1.363 


3.536 


1.000 


-10.561 


7.835 


2 


1 


-14.462* 


4.889 


.023 


-27.180 


-1.744 


3 


-15.825* 


3.930 


.002 


-26.050 


-5.600 


3 


1 


1.363 


3.536 


1.000 


-7.835 


10.561 


2 


15.825* 


3.930 


.002 


5.600 


26.050 



Based on estimated marginal means 



*. The mean difference is significant at the .05 level, 
a. Adjustment for multiple comparisons: Bonferroni. 



The participants scored significantly lower on test two even though it assessed comprehension of a book delivered in 
the exact same format as the book for test 3. Test one assessed comprehension of a print book, and test three 
assessed the comprehension of an e-book. However, the scores for test 1 (print book) compared with test 3 (e-book) 
were nearly identical. Test items were written using standardized tests typically used in the subjects’ classroom. 

Unlike the test items, the survey items were intended to measure the level of enjoyment that the participants 
experienced while reading the books in the various formats. Additionally, Chi-square cross tabulations were 
conducted to reveal any interaction among the four surveys. The first three surveys measured enjoyment of each 
book, and the fourth survey measured their preference for reading books in either print or electronic format. The 
only significant interaction among the first three surveys was the question regarding enjoyment of the books. The 
participants indicated an equal level of enjoyment regardless of the book format. 
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The survey items regarding ease, self-assessed comprehension, motivation, self-selected reading, and 
recommendations did not yield any significant interaction results. The final survey measured their preference for 
reading books in either of the two formats. Most of the students indicated that they would prefer to and would 
continue to read books in either format. The students showed no particular preference for reading the books in either 
format. There is one item of interest-- subjects indicated that they were impressed with reading the wide selection of 
e-books and the various features of the computer program. They also indicated that they would like to log into the 
Raz-Kids website from home and read books from home. The students did not have difficulty with navigating the 
books on the computer, and they did not find it confusing. 

Nonparametric analysis. 

Survey 1 measured the enjoyment for Yellow House, and Survey 3 measured enjoyment for Sweet Potato. 
In terms of the enjoyment question on Survey 1 compared to Survey 3 Sweet Potato, the response to Enjoy is 62.5% 
for Yellow House compared to 71.4% for Sweet Potato. See also the response to somewhat true at 25% for Yellow 
House compared to 33.3% for Sweet Potato. This shows more students responded at a higher level of enjoyment for 
Sweet Potato e-book than for the level of enjoyment for Yellow House print book. 

Table 3. Compare surveys on enjoyment for Yellow House print book with enjoyment for e-book Sweet Potato. 
Survey 1 Y ellow House Mystery Print version “enjoy” with Survey 3 Sweet Potato e-book “enjoy. 



Compare Survey 1 with Survey 3 


Not True 


Somewhat 

True 


True 


Total 




True 


Count 


1 


2 


5 


8 






Expected Count 


1.5 


3.0 


3.5 


8.0 






% within Survey 1 
Enjoy 


12.5% 


25.0% 


62.5% 


100.0% 






% within Survey 3 
Enjoy 


33.3% 


33.3% 


71.4% 


50.0% 








6.3% 


12.5% 


31.3% 


50.0% 



Compare this to Table 4 as shown below. Students’ response to the enjoyment question for Mystery Wind compared 
to enjoyment question for Yellow House is the same. The format print book compared to e-book does not appear to 
be a factor. This leads us to the conclusion that content, theme, setting, and plot may have a stronger effect on 
children's reading motivation than format. 

Table 4. Compare student response to enjoyment for Yellow House book compared to Mystery Wind book (e-book). 
Students respond to survey enjoyment of print book at the same rate as e-book enjoyment. 



Survey 1 to Survey 2 Enjoy 


Not True 


Somewhat 

True 


True 


Total 


Survey 1 to Survey 2 Enjoy 


True 


Count 


1 


0 


6 


7 


% 

within 
Survey 1 
Yellow 


14.3% 


.0% 


85.7% 


100.0% 
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Enjoy 










% 

within 

Survey 2 
Mystery 
Enjoy 


25.0% 


© 

N® 

o N 


85.7% 


46.7% 


Total 


6.7% 


.0% 


40.0% 


46.7% 



In Table 5., Chi Square reports a significant relationship between students' response to Survey 3 Sweet Potato 
enjoyment with Survey 4 Choice (r = 12.97, p < 01, 2-tailed). The variable Choice refers to being able to choose 
from many book titles. There is a strong correspondence in survey response between those students who enjoyed the 
e-book Sweet Potato with students' who liked being able to choose from a wide selection of book titles. This 
suggests that students who enjoyed the e-book Sweet Potato are motivated when given the opportunity to choose 
their own reading materials. 

Table 5. Chi Square analysis showing relationship between student response to enjoyment of e-book Sweet Potato 
with student response to being able to choose own reading materials. 



Chi-Square Tests for Pearson correlation between “Enjoy” Sweet Potato with “choice of 

titles” 




Value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 


12.974 3 


4 


.011* 


Likelihood Ratio 


9.709 


4 


.046 


Linear-by-Linear Association 


4.532 


1 


.033 


N of Valid Cases 


18 







Reading motivation and engagement are enhanced when students have choice in reading materials. These 
findings are consistent with the findings of Flowerday, et. al. (2004). The wide selection of titles possible through 
online e-book websites is motivational for children. This is similar to the freedom of choosing any book from the 
classroom or school library collection. One benefit in use of e-books is improved access to a greater number of 
reading selections. Currently most e-books for K12 classrooms are limited to subscriptions sites with several 
restrictions. First, in most schools the district technology director must grant privileges for download of e-books to 
any particular school server. This is a minor obstacle but does require time and effort on the part of the teachers. In 
addition, the authors discovered very few current titles available online, paid or free. Older book titles that do not 
require copyright are available in abundance. Most children in this study, however, preferred books that relate to 
their own culture and environment. Finally, Chi Square showed a perfect correspondence between enjoyment scores 
for the print book Yellow House compared to the e-book Sweet Potato. This suggests that format is not important as 
the content, theme, and general writing style of the book. Although students responded favorably to such features as 
popup windows with definitions and word pronunciation, this study showed no significant effect on students' 
reading comprehension. Earlier research in multimodal learning reports positive effect on learning when more than 
one modality is used for reading instruction. Why then, was there little effect on reading comprehension scores with 
multimedia interactive e-books used in this study? This opens the door for continued work in how special features in 
e-books might enhance reading motivation and ultimately comprehension. 
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Discussion. 



The results of the ANOVA yielded interesting results since the students scored significantly lower on the 
second test based on a book that was read in the same electronic format at the third book. The survey results also 
indicated that the participants enjoyed the first and third books better than the second book. The mean enjoyment 
score for book one was 2.32 which was identical for the mean enjoyment score for book three. The mean enjoyment 
score for book two was 2.1 1 which was significantly lower. This coincidently, coincided with the significantly lower 
score on the comprehension test for book two, while the comprehension scores for books one and three were nearly 
identical. These findings would indicate that the format of the book did not matter as much as the level of enjoyment 
that the children received from the storyline. All three books were read in the same fashion with the children “bump 
reading” in the same reading groups for each session. The groups also had the same facilitator for each session. The 
groups also all engaged in the same extension activities after reading each selection. The homogeneous groupings 
also ensured that their reading levels were comparable. The only difference in the study design was the format of the 
books. Readings of the print book and the second e-book resulted in comparable amounts of enjoyment and 
comparable comprehension scores. 

The first reading selection was the first three chapters of The Yellow House Mystery. This is the third book 
in a series of mysteries about four children who live with their grandfather and fashion themselves as young 
detectives. In this book, the children must solve the mystery of a man who disappeared long ago while living in a 
house on their grandfather’s property. The first three chapters of the book were read in groups, and participants were 
subsequently tested on those three chapters. The remainder of the book was read aloud to the students with the 
teacher reading a chapter a day after recess. 

The third book that the participants read in electronic format was entitled The Sweet Potato Challenge. This 
e-book was about a cooking contest and contained recipes for various desserts. This book centered on children living 
in America and challenging each other to make the best sweet potato pie. The children “bump read” this entire book 
and were also able to click on highlighted vocabulary words to either have the word pronounced or have the 
definition displayed. The facilitators ensured that the only child who was allowed to use the interactive features was 
the child who was reading aloud. The book also contained a feature that would allow children to change pictures 
without changing the page. Therefore, some children click the “enter” button expecting the page to change while 
only seeing a different picture. The text remained the same. In responding to the survey, children did not find this 
feature difficult or annoying. 

The second book was also read in electronic format. This book was entitled The Mystery Wind. Again, the 
research design remained the same. Children bump read and used the interactive features. One difference was that 
this was the first book that the children read using the electronic format. They were allowed to explore and the other 
books in the Raz-Kids site before reading The Mystery’ Wind, but this was the first e-book that the children had ever 
read. Another notable difference between this book and the other two was that The Mystery’ Wind was about a little 
girl in Africa who discovers a mysterious wind that brings good fortune to her village. The program offered sound 
effects of the wind blowing and had other similar features to the third book that the participants read in electronic 
format. 



The most notable difference between this book and the other two was that this book was not set in America 
and did not contain a setting that was automatically familiar to the participants. The huts in which the characters 
lived and the oxen that roamed around the village may not have been as easy to understand as the settings of the 
other two books. The students also did not enjoy the story as indicated on the survey. Their lack of enjoyment was 
also reflected in their comprehension of the score. It is possible that their reading engagement was lower because of 
their lack of interest in the story. 

A central finding in these data was the strong correlation between enjoyment of the final e-book that the 
children read and their preference for a choice of books. As Flowerday, et. al., (2004) found, children are highly 
motivated to read and remain engaged in literacy activities when afforded a choice of what to read. The participants 
in this study all rummaged through the Raz-kids cite to find other books which might interest them when given the 
opportunity to do so. This correlation between enjoyment of Sweet Potato, an e-book, and a preference for self- 
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selecting reading material suggests that the electronic format combined with the opportunity for choosing books was 
a highly motivating factor for children to read. 

According to data from the fourth survey, the participants had no preference for reading in either format. 
Over half of the participants wanted to continue to read both print and e-books. Only three participants indicated that 
they would only like to read e-books in the future, while just one participant indicated a future preference for print 
books only. Three participants also indicated no preference at all for either format. The interactive features of the e- 
books did not sway them to a desire for using electronic formats as their sole source reading. The observation data 
revealed that the children clearly displayed an interest in the interactive features such as having the books read to 
them, having vocabulary words pronounced for them, viewing various pictures, and sound effects. The data also 
revealed that they thoroughly enjoyed roaming through the vast selection of e-books and reading selections of their 
own choosing. However, they still indicated that they would like to continue reading books from both formats. 

The results from both the comprehension tests and the surveys indicated that the format in which children 
read the material is not an important indicator for this study. The e-book format did not significantly increase 
comprehension, enjoyment, or engagement. The data clearly indicate that children prefer to have a choice of reading 
material and that the format was not as central to reading engagement as a connection with the story’s characters and 
setting. As publishers and teachers begin to introduce electronic reading to younger children, strong consideration 
must be given to the quality and quantity of the books provided for children to read. The results of this study 
indicated that the format in which the book was delivered did not matter as much as the suitability of character, 
theme, and setting of the books and how these align with personal preferences of the reader. A second outcome is 
further evidence which suggests a wide variety of reading choices and the opportunity to select books does impact 
reading engagement and ultimately reading comprehension. 
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Abstract 

The purpose of this study was to examine the changes in self-efficacy of student teachers engaged in a 
technology-supported instructional program as part of their field experience. The program was designed to introduce 
the elements of cognitive apprenticeship (modeling, scaffolding, coaching, reflecting, and community building) into 
the supervision process. Eighteen participants completed a pre- and post- measure of self-efficacy. After controlling 
for differences in their experience with the cooperating teacher, ANCOVA on post-efficacy mean scores with pre- 
scores as a covariate indicated a statistically significant difference favoring those in the program. Interviews were 
conducted to determine the role that technology played in building efficacy knowledge and engaging with the 
sources of self-efficacy. Implications for creating technology-enhanced cognitive apprenticeships and improving the 
supervision of student teachers during the field experience are discussed. 

Introduction 

Teacher self-efficacy - that is, a teacher’s belief in his or her ability to achieve specific results in a given 
context (Tschannen-Moran, Hoy, & Hoy, 1998) - plays a role in a teacher’s performance. A teacher’s efficacy 
beliefs affect what they are willing to attempt in the classroom and how persistent they will be at succeeding as a 
teacher (Mulholland & Wallace, 2001; Tait, 2008; Tschannen-Moran & Hoy, 2001). Tschannen-Moran et al. (1998) 
describe the interplay between self-efficacy and performance as cyclical. For example, a positive teaching 
experience can help improve a teacher’s beliefs in his or her abilities. This, in turn, increases the likelihood of more 
positive experiences in the future. Similarly, a negative experience might lower a teacher’s beliefs in his or her 
abilities, which would increase the likelihood of more negative experiences in the future. 

Bandura (1997) identified four sources of efficacy knowledge: mastery experiences, vicarious experiences, 
verbal persuasion, and physiological arousal. In the context of student teaching, the first three sources have been 
thoroughly established by several researchers. For example, mastery experiences are described as teaching tasks that 
inform a teacher’s knowledge of his/her own abilities, such as regularly and successfully planning and implementing 
instruction (Knoblauch & Hoy, 2008; Tschannen-Moran & Hoy, 2007). Vicarious experiences are described as the 
observation of modeled instruction, including observing one’s self, or self- modeling, and learning about the 
experiences of others (Knoblauch & Hoy, 2008; Mulholland & Wallace, 2001). Verbal persuasion entails receiving 
feedback about one’s performance or emotional support from colleagues, administrators, students, or the greater 
community (Liaw, 2009; Tschannen-Moran & Hoy, 2007). Mastery experiences that are informed by the other two 
sources of efficacy are suggested to be the most likely to shape a teacher’s efficacy beliefs (Bandura, 1997; 
Knoblauch & Hoy, 2008; Mulholland & Wallace, 2001). 

The field experience is an opportune time to positively influence and shape novice teacher efficacy (Liaw, 
2009; Mulholland & Wallace, 2001). Student teachers frequently engage in activities that are believed to improve 
their self-efficacy, such as observing a variety of competent models (Knoblauch & Hoy, 2008), sharing experiences 
with others (Liaw, 2009; Main & Hammond, 2008), and receiving recognition for their successes (Mulholland & 
Wallace, 2001; Tschannen-Moran & Hoy, 2007). They also have ample access to all three sources of efficacy 
knowledge - mastery experiences, vicarious experiences, and verbal persuasion - at a time when they are most open 
to them. Tschannen-Moran and Hoy (2007) found evidence of this in a regression analysis that included contextual 
factors, verbal persuasion, and mastery experience as predictors of self-efficacy. In that study, both mastery 
experiences and verbal persuasion made a significant contribution to the efficacy beliefs of 75 novice teachers, 
whereas only mastery experiences made a significant contribution to the efficacy beliefs of the remaining 180 
experienced teachers. 

Technology may play an important role in positively impacting the self-efficacy of novice teachers. The 
activities that cultivate self-efficacy, such as guided interaction with experts (Tschannen-Moran & Hoy, 2007) and 
receiving ample support (Capa & Woolfolk-Hoy, 2005), have been achieved through technological interventions. 
Researchers have used telecommunications technology (Barnett, Keating, Harwood, & Saam, 2002; Hew & 
Knapczyk, 2007), electronic performance support systems [EPSS] (Barnett, et al., 2002; Hew & Knapczyk, 2007; 
Liu, 2005; Wild, 1998), and video (Bannink, 2009; Sherin & van Es, 2005) to increase interaction and support 
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between novices and expert teachers so that novices receive more frequent support during teaching tasks. However, 
prior studies on the use of technology with novice teachers focus more on participant perceptions of the tool and less 
on student outcomes such as efficacy (Gentry, Denton, & Kurz, 2008). 

The purpose of this study was to examine the teaching self-efficacy of students who participated in a 
Technology-Enhanced Student Teacher Supervision [TESTS] throughout their field experience. TESTS the 
traditional field experience using telecommunications technology, EPSS, and video reflection to support the 
cognitive apprenticeship between novice and expert teachers. The research questions were: 

1. How does the self-efficacy of student teachers who participate in TESTS differ from students who do not? 

2. What role do TESTS technologies play in cultivating self-efficacy? 

3. Are the student teacher’s reported levels of self-efficacy reflected in their actual performance? 

Method 

Instructional Program 

The TESTS instructional program is a series of five online learning modules that incorporate 
telecommunication technologies, video, and EPSS during the field experience. In order of completion, the five 
modules were Analyzing the Teaching Context, Classroom Management, Planning Instruction, Engaging the 
Learner, and Assessing the Learner. The instructional activities in those modules were designed around the elements 
of cognitive apprenticeship (Collins, Brown, & Holum, 1991). Prior evaluation of TESTS suggested a number of 
positive results (Alger & Kopcha, 2009, in press). 

The instructional activities found within those modules are summarized here: 

1 Asynchronous discussion. Triad members (student teachers, cooperating teachers, and university 
supervisors) use the online discussion boards to share experiences and/or receive feedback and advice on 
relevant classroom issues, such as classroom management, instructional strategies, and assessment 
methods. Student teachers are asked to post to the discussion boards on a bi-weekly and are encouraged to 
write about topics they find relevant. Experts in TESTS regularly read and often reply to the postings of 
student teachers other than their own. 

2 Guided observation including video reflection. Student teachers in TESTS are observed a total of six times 
per semester; three times by the supervisor and three times by the cooperating teacher. When they are 
observed, the observers use a downloadable form that is completed electronically and then shared with the 
others in the triad. The form contains a variety of Likert-type and open-ended items that focus the 
observation on the planning, instruction, assessment, and reflection [PIAR] of student teachers. In addition, 
student teachers videotape and observe themselves delivering a lesson once per semester. The self- 
observation is completed using a guided reflection form and both the video and reflection are reviewed 
with an expert for feedback. 

3 Lesson planning. Student teachers regularly create lesson plans using an Electronic Performance Support 
System [EPSS]. Student teachers using the EPSS complete an online form that contains key reminders of 
how to plan effective lessons. They then submit lessons electronically to the cooperating teacher and 
supervisor for feedback. Each lesson stored and published to an online searchable database, and all 
members in TESTS can use the database to search for and view student lessons at any time. 

Participants 

Eighteen student teachers participated; half (9) received the TEIP. Both groups were geographically placed 
in urban districts with similar demographics that included a large proportion of English Language Learners and high 
numbers of students eligible for free or reduced lunch. 

All students at this university engage in two semesters of the field experience and participate in the same 
university coursework. Both groups were beginning their second semester of student teaching; students in TESTS, 
however, were placed in a new classroom with a new cooperating teacher, whereas the non-TESTS group remained 
with the same classroom and cooperating teacher. Students engaged in TESTS were trained to use the required 
technology tools. 



Measures 

1 Teachers' Sense of Efficacy Scale [TSESJ. The primary measure of teacher self-efficacy was the short form 
(12 items) of the Teachers' Sense of Efficacy Scale [TSES] (Tschannen-Moran & Hoy, 2001). Student 
teachers reported on how much they felt they could do with regard to a number of common teaching tasks 
(class management, student motivation, delivery of content) on a nine-point scale with descriptors at every 
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odd number, from 1 (nothing) to 3 (very little) to 5 (some influence) to 7 (quite a bit) to 9 (a great deal). A 
sample item from the measure is, “How much can you do to control disruptive behavior in the classroom?” 
The short form of the TSES has been established as a valid and reliable measure of teacher efficacy beliefs 
(Fives & Buehl, 2009). The reliability of this instrument was .87 in this study and has been reported at .90 
or higher in prior studies (Fives & Buehl, 2009; Tschannen-Moran & Hoy, 2001). 

2 Learning to Teach Scale. The original survey, which contained 10 items, measured both the level of 
guidance from the cooperating teacher and the amount of imitation the student teacher engaged in. Student 
teachers in this study reported on the 5 items associated with the level of guidance they received from their 
cooperating teacher. Responses were given on a six-point scale with the following descriptors: never (1), 
almost never (2), sometimes (3), often (4), almost always (5), and always (6). A sample item from the 
measure is, “My cooperating teacher offers me guidance to improve my teaching.” The scale has been 
established as both valid and reliable. The reliability of this measure in this study was .93. 

3 Student Teacher Interview Protocol. Student teachers reported on the changes in confidence they 
experienced with regard to common teaching practices (planning, implementing, and assessing instruction) 
over the semester and the role technology played in facilitating those changes. Sample items from the 
protocol are, “In what ways did technology have an impact on your ability to: Plan lessons? Implement 
them? Assess them? Reflect on your teaching?” and “Do you think that the technology you used as part of 
learning to teach helped or hindered your growth this semester? Why / why not?” The interviews were 
semi- structured and lasted an average of 20 minutes. 

Procedures and Timeline 

One cohort of 18 student teachers received the intervention of TESTS and the other cohort of 20 teachers 
did not. The cohort that received TESTS was selected because the instructor of the seminar for that cohort’s field 
experience volunteered to use TESTS. Students, however, were assigned to TESTS in a random fashion. All 
students in the entire secondary single subjects credential program at this university were placed randomly into one 
of four cohorts at the beginning of the year. In this way, students were equally likely to be placed in the cohort that 
received TESTS or in the comparison group. 

Student teachers in the TESTS group engaged in the instructional program over the course of two 
semesters. Because they participated in TESTS as part of an on-campus seminar course that met once per week, the 
seminar leader set the pace for completing the activities in TESTS. The seminar leader also participated in online 
discussions and was responsible for monitoring and promoting expert participation in TESTS. Non-TESTS students 
participated in a similar seminar course for their own cohort; however, this seminar had no instructional program 
like TESTS as part of the coursework. Instead, they participated in a series of face-to-face, in-class discussions that 
focused on resolving issues that were specific to the cohort. 

Although the participant pool consisted of 38 student teachers, only 18 volunteered to participate in our 
study. Student teachers were solicited for participation at the beginning of their second semester and completed the 
TSES at that time as a pre-measure. One week prior to the end of the second semester, they once again completed 
the TSES as a post-measure, and also completed the Learning to Teach Scale. Both student and cooperating teachers 
participated in telephone interviews shortly after student teachers completed the surveys; the interviews were 
recorded and transcribed for analysis. 

Design and Analysis 

TSES post-survey mean scores were analyzed between eSupervision and non-eSupervision groups using 
ANCOVA with the pre-survey as a covariate. ANCOVA was used to correct for initial differences (Iacobucci, 

2001); eSupervision students may have had different levels pre-efficacy due to the fact that they were placed in a 
new classroom with a new cooperating teacher. Learning to Teach mean scores were compared using ANOVA to 
rule out differences due to the relationship with the cooperating teacher. 

Because of the small sample size, student teacher interview data were analyzed by case to add validity to 
ANCOVA results (Yin, 1994). They were analyzed in a qualitative manner, organizing responses into common 
themes. Cooperating teacher interview data were examined to determine whether student teachers accurately 
described their changes in confidence in the light of their cooperating teacher’s observations of their performance. 

Results 

The mean scores for the Learning to Teach Scale by group fell between the rating of “often” and “almost 
always”. Specifically, the scores were 4.70 for the TESTS group and 4.29 for the non-TESTS group, a non- 
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significant difference. These scores indicated that student teachers from both groups experienced similar interactions 
with their cooperating teachers, and with similarly high frequency. 

Analysis of covariance [ANCOVA] on the 18 post-TSES scores with pre-TSES scores as a covariate 
revealed a statistically significant difference between the TESTS (M= 7.06) and non-TESTS groups (M= 6.77), 

F(l, 16) = 7.67, p<. 05, 2 = .35. 

A total of 9 student teachers (5 TESTS, 4 non-TESTS) were interviewed to give them the opportunity to 
describe the role that technology played in their student teaching field experience and in developing their teaching 
skills. Both TESTS and non-TESTS students were included in the interview sample to determine what technologies 
both groups did use and any commonalities with regard to how those technologies were used. 

Both groups reported using a variety of technologies to help them learn to teach during their field 
experience. Both TESTS and non-TESTS participants reported using email and video reflection; TESTS participants 
also reported using discussion boards, the online lesson plan EPSS, and observation templates. The results 
associated with email, the video reflection, the lesson plan EPSS, and the observations forms are summarized in 
Table 1 and are presented in detail below. 

Electronic mail. Students from each group used email with similar frequency and for similar purposes. All 
stated that they infrequently used email with their supervisor, primarily to arrange site visits and triad meetings. 

With regard to the cooperating teacher, email contact varied in frequency from daily (1 TESTS, 1 non-TESTS) to 
weekly (2 TESTS, 3 non-TESTS) to only several times a semester (2 TESTS, 1 non-TESTS). These varied in nature 
from planning to classroom management issues to the organization of the school. The two participants who used 
email daily with the cooperating teacher were the only two that reported using it to receive advice about lesson plans 
and share classroom experiences with the cooperating teacher. 

Video reflection. Students in both groups used video technology to record themselves teaching a lesson and 
review the videotaped lesson. Students in the TESTS group used a standard protocol to reflect on the lesson, and 
then received feedback on the videotaped lesson from their supervisor. All TESTS students found that the feedback 
they received was a powerful source of information about their ability to implement instruction. 

The video experience afforded TESTS students with two opportunities to improve their efficacy. First, the 
act of observing themselves functioned as a vicarious experience through self-modeling, which created an 
opportunity to challenge their own thinking about their level of mastery with regard to teaching. One described: “I 
was like, ‘Wow! My kids are being really well-behaved and that’s something really smart that this kid just said, and 
I don’t even know if I acknowledged that he said it’.” Another student noted that the observation template helped 
guide this self-modeling in a purposeful manner, stating, “I tend to be hard on myself. So having specific areas [of 
PIAR] to watch for, and thinking, ‘Oh, 1 think I’m actually ok with doing this’ makes me more confident as a 
teacher”. 



The second opportunity to improve their efficacy came from the feedback from the supervisor provided 
student teachers, which acted as a form of verbal persuasion that informed their level of confidence in their ability to 
teach. One student explained, 

I think what [the video reflection] helped most with was that [my supervisor] pointed out things that I missed even 
watching it. He saw a different perspective but he came back with a few things and I was like, ‘Wow! I didn’t notice 
this before.’ 

All non-TESTS students videotaped themselves teaching. Similar to the TESTS group, watching the video 
provided non-TESTS students with an opportunity to engage in a vicarious experience of self-modeling. They were 
able to observe their own performance and gauge their success with their own teaching ability. One student 
explained, “I could see the strategies [I was using] and see that they were working.” Unlike the TESTS group, 
however, they did not have an opportunity to receive feedback, or verbal persuasion, from an expert but rather from 
peers in a large-group setting. They reported that the feedback helped them identify and change some issues in their 
own teaching, but felt that overall it was superficial and uninformative. 

Lesson Plan EPSS. TESTS student teachers were required to post daily lesson plans using the Lesson Plan EPSS. 
This appears to have acted as a mastery experience for the TESTS participants, who all reported that the repeated 
use of this tool helped them internalize and master the process of planning. One student stated, “While 1 hated 
[planning daily lessons], it actually turned out to be a good thing because I felt I got better and better at it as I did it 
more and more.” 
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Table 1 



Interviewed students ’ use and impact of technology’ on self-efficacy by group 



Technology 


eSupervision 


Non-eSupervision 


Electronic 

Mail 


l Likely to have had similar impact as 
non-eSupervision due to similar 
frequency and purpose 




Video 

Reflection 


Received feedback from supervisor 
Provided a vicarious experience of 
self-modeling that informed level of 
mastery 

Verbal persuasion regarding 
performance was a powerful source of 
efficacy information 


Received feedback from peers 
Provided a vicarious experience of 
self-modeling that helped identify 
issues with teaching 
Student teachers felt the verbal 
persuasion they received was 
superficial and uninformative at 
times 


Lesson Plan 
EPSS 


Repeated use of tool acted as a 

mastery experience 

EPSS less useful as ability to plan 

improved 


l Lacked any formal instruction with 
regard to planning 


Discussion 

Boards 


n Expert comments prompted reflective 
practice 

l Experiences shared online 

provided vicarious experiences that 
informed own level of teaching 
mastery 


l No discussion boards used 

to support the field experience 


Cooperating 

Teacher 

Observation 

Forms 


l Cooperating teacher formally 

observed student teacher 3 times a 
semester with an observation form 

l Provided useful and specific 

feedback 

l Created opportunity to 

receive verbal persuasion and inform 
mastery about PIAR 


l No formal observation by 

cooperating teacher 



It was clear, however, that as mastery levels increased, student teachers felt the need to use the tool less. 
One student summarized this trend: 

It helped me to really get the hang of the whole ‘planning every day’ and everything. By second semester, 1 had 
already gotten the idea of how it flowed and it was just an extra thing that I didn’t participate in much. 

In contrast, the non-TESTS group reported that they lacked any formal mechanism for learning to plan lessons and 
relied heavily on the cooperating teacher for assistance with this task. Several noted that this was due to a lack of 
instruction at the university level regarding the planning of lessons. 

Discussion boards. TESTS students reported using the discussion boards to support their learning to teach 
during the field experience; in contrast, non-TESTS students did not report using any type of discussion board 
related to the field experience. For TESTS students, the discussion board functioned as an opportunity to use 
vicarious experiences to judge their own level of mastery with regard to teaching. One student stated, “I really 
wanted to know if other people were having the same issues that I was, and [the discussion board] was a huge way 
to communicate that.” Another similarly noted, “I appreciated seeing things that were going on with other student 
teachers and seeing they were in the same place as me.” Similar to the lesson plan EPSS, TESTS students reported 
using the discussion boards as a tool for measuring their own abilities more when they were less confident, and less 
when they became more confident. 
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Cooperating Teacher Observation Forms. The use of the observation form by the TESTS group created a 
formal and guided opportunity for student teachers to improve their self-efficacy through verbal persuasion. 
Discussion guided by the observation form provided valuable information that informed their level of mastery with a 
specific focus on PIAR and helped them discern the positive attributes of their performance from the negative ones. 
One student concisely summarized this, stating, “I tend to be hard on myself, so having specific areas where I know 
I’m actually doing ok with something makes me more confident.” 

Discussion 

TESTS students scored significantly higher on the TSES at the conclusion of their field experience than 
non-TESTS students. This difference is likely due to the manner in which TESTS students used technology during 
the field experience. TESTS students had greater access to positive feedback from a greater variety of experts and 
novices than non-TESTS students due to technology. They received both advice and support through the discussion 
boards - often from a supervisor that was not their own or from any number of peers. In addition, they were 
formally observed by and received feedback from their cooperating teachers in addition to their supervisors. It may 
be that the difference in self-efficacy between the two groups is also due to the fact that TESTS students received a 
greater variety of positive feedback during the field experience. This would be consistent with others (Bates & 
Khasawneh, 2007; Wu & Lee, 2004) who found that feedback from experts played a role in shaping the efficacy 
beliefs of preservice teachers in online settings. It also supports researchers who suggested that the performance and 
attitudes of student teachers would be improved by connecting novices and experts using technology (Joia, 2001; 
Owston, Wideman, Murphy, & Lupshenyuk, 2008; Wang & Bonk, 2001). 

TESTS students also had more opportunities to inform their level of mastery with many teaching tasks due 
to their use of technology. For example, they reported that their ability to plan lessons became both internalized and 
routine due to the Lesson Plan EPSS, indicating a high level of mastery associated with the skill of planning. They 
reported reading about the experiences of others on the discussion boards, a form of vicarious experience, which 
they used to judge their own level of success with several aspects of teaching. They also received feedback from an 
expert on their videotaped lessons that informed their perception of mastery associated with implementing 
instruction. These experiences were likely to have had a powerful impact on the self-efficacy of TESTS students. 
Mastery experiences that are informed by other sources of self-efficacy are noted as one of the most powerful 
influences on a teacher’s level of efficacy (Bandura, 1997; Mulholland & Wallace, 2001; Tschannen-Moran, et ah, 
1998). Others have similarly found that student teachers have used feedback from others to evaluate their own 
ability as a result of using discussion boards (Liu, 2005), felt more confident with regard to planning tasks due to 
using EPSS (Hacker & Sova, 1998; Wild, 1998), and were better able to identify and change issues with their own 
teaching as a result of video reflection paired with expert feedback (Lee & Wu, 2006). 

While the cooperating teacher clearly plays one of the largest roles in shaping student teacher self-efficacy 
(Knoblauch & Hoy, 2008), the results of this study suggest that an instructional program like TESTS can be 
effective at augmenting that influence. Teacher educators interested in using technology to support the supervision 
of student teachers during the field experience should note that the primary purpose of TESTS is to support a 
cognitive apprenticeship - technology is merely the vehicle for accomplishing this task. The activities that student 
teachers completed in this study, such as guided observation and feedback, online discussion of issues among peers 
and experts, lesson plan EPSS, and video reflection, are likely to be ways to support the triad during the field 
experience. This supports others (Bates & Khasawneh, 2007; O'Neill & Harris, 2004; Owston, et ah, 2008) who 
have noted that learning outcomes are likely to improve for students in online settings when the goal is to use 
technology to connect them more purposefully with experts. 

The most noticeable limitation of this study is the small sample size. However, there are several reasons to 
believe that similar results would occur with a different group of students. The interview data support the statistical 
findings with regard to teacher self-efficacy - TESTS students engaged more frequently with a variety of sources of 
self-efficacy knowledge. Researchers have found that technology can improve the quality of student teacher 
supervision and improve the performance and attitudes of teachers learning to teach (Liu, 2005; Pianta, Mashburn, 
Downer, Hamre, & Justice, 2008). The results of this study are consistent with those findings, suggesting that they 
are valid in light of the small sample. 

This paper was an initial attempt to examine the impact of technology on both teacher performance and 
self-efficacy in a single context. While the results are encouraging, the sample size limits any future research should 
begin with a replication study that improves the size of the sample and thus the power of the statistical analysis. 
Other research could include examining the individual influences on teacher performance and self-efficacy within a 
technology-rich learning environment such as TESTS. Both technological and non-technological influences should 
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be examined to determine which effectively shape teacher performance and self-efficacy, and, more importantly, to 
what degree. Such studies would help us further develop effective strategies for preparing teachers for the classroom 
and address the need noted by several researchers to examine the connection between efficacy and performance 
(Labone, 2004; Wheatley, 2005) and the impact of technology on both (Gentry, et al„ 2008). 
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Abstract 

This case study examined three structural attributes observed in students’ causal maps (total links, temporal 
flow, horizontal location of outcomes nodes) and their relationship to the accuracy of students’ maps (number of 
correct root causes, number of root cause links) to determine the attributes that should be emphasized during map 
construction. The findings from regression analyses suggest that increasing temporal flow can substantially increase 
accuracy in number of correctly identified root causes, and placing limits on the total number of causal links can 
increase the number of correctly identified root cause links. Implications of these findings on how to manipulate the 
causal mapping task and tools and directions for future research are discussed. 

Introduction 

Causal maps, a network of nodes and links that define the causal relationships between nodes, can be used 
in science education as a tool to teach and assess learners’ systemic understanding of complex problems and 
phenomena (Ruiz-Primo & Shavelson, 1996). Given that causal maps in theory represent learner’s cognitive 
structures, their complex reasoning, and conceptual development (Jonassen, 2008), causal maps can be and have 
been used to elicit, articulate, refine, assess, and improve understanding, analysis, and the identification of the 
causes and causal mechanism underlying complex problems. Improvements in students’ understanding have been 
observed particularly when students work both individually and collaboratively to construct their own maps as 
opposed to simply presenting students the instructor or expert maps (Nesbit & Adesope, 2006). Maps can be used to 
support collaborative learning when students compare their maps to identify, trigger, and focus group discussions 
around key differences in viewpoints and understanding (Jeong, 2009 & 2010a). 

A growing number of studies on causal maps and other types of maps (e.g., concept maps) have formulated 
various metrics to measure the accuracy and structural attributes of students’ maps (parsimony, temporal flow, total 
links, connectedness) - particularly attributes believed to be correlated to map accuracy and attributes that can be 
potentially used to generate guidelines or constraints to help students create more accurate maps (Scavarda et al., 
2004; Ifenthaler, Masduki & Seel, 2009; Jeong, 2009; Plate, 2010). Studies have been conducted to determine how 
different constraints imposed on the map construction process affect student’s maps and learning - constraints like 
imposing hierarchical order by allowing students to move and re -position nodes (Ruiz-Primo et al., 1997; Wilson, 
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1994), providing terms for nodes (Barenholz & Tamir, 1992), providing labels for links (McClure & Bell, 1990), 
and allowing more than one link between nodes (Fisher, 1990). 

In addition, studies have been conducted to develop software tools to automate and reliably measure both 
the accuracy and the structural attributes of maps. Software programs like HIMATT (Ifenthaler, 2008) and jMAP 
(Jeong, 2010b) are being used to address issues of rater reliability and validity by using software to automate 
measurements that can be used to test the correlation between different structural attributes and accuracy of 
students’ maps (Ifenthaler, Madsuk & Seel, 2009), and to measure how maps change over time and how observed 
changes over time contribute to convergence in shared understanding between learners (Jeong, 2010a). 

However, students’ maps can vary widely in accuracy when maps are compared to expert maps (Ruiz- 
Primo & Shavelson, 1996; Scavarda et al., 2004). The critical question here is whether the variance in accuracy is a 
reflection of students’ lack of knowledge and understanding of the topic under study, or is it more a reflection of 
student’s lack of understanding and skills with drawing causal maps? Based on their review of the empirical 
research, Ruiz-Primo & Shavelson (1996) concluded that maps (including the assessment rubrics) should not be 
used in the classroom for large-scale assessments until students’ facility, prior knowledge/skills with using maps, 
and associated training techniques are thoroughly examined. In other words, researchers must examine how 
differences in causal mapping skills and processes lead to differences in map accuracy between students with equal 
knowledge and understanding of the concepts/problems they are trying to articulate with causal maps. Ruiz-Primo et 
al. (1997) also found in their study that requiring students to hierarchically structure their maps did not produce any 
gains in the match between students’ and experts’ maps. However, Ruiz-Primo at al. conceded that they had 
difficulties in developing a clear operational definition and measure of hierarchical structure. As a result, more 
studies are needed to developing and articulating measures of hierarchical structure and how such measures 
ultimately reflect students’ level of learning and understanding. 

Given all of the above, new research is needed to: a) identify tendencies and potential weaknesses in the 
way students construct causal maps with minimal or no prior instruction on causal mapping; and b) determine to 
what extent each noted weakness affects the accuracy of students’ maps while controlling for students’ level of 
knowledge and understanding of the concepts/problems they are trying to articulate via causal maps. A clear 
understanding of the weaknesses and their effects will provide the foundation on which to identify the most 
appropriate guidelines, constraints, and interventions for improving the map construction process and 
quality/accuracy of students’ causal maps. 

Using the case study method, this study examined the accuracy of students’ maps based on the ratio of 
correctly/incorrectly identified root causes. This study also examined accuracy in terms of total number of correctly 
identified root links (links stemming from root causes) to gauge how well students understand the causal chains, 
mechanisms, and mediating factors underlying cause-effect relationships between root causes and outcomes. These 
measures were tested for their correlations with three attributes: total number of causal links ( total links), ratio of 
right/left pointing links (temporal flow), and distance of outcome node from left edge of screen ( location of the node 
representing the final effect/outcome). 

The purpose of the study were to determine to what extent do these three attributes (number of links, 
temporal flow, and location of outcome node) are correlated with (and possibly contributes to) level of accuracy. 

The findings can then be used to identify which attributes to emphasize to students by imposing specific constraints 
within the causal mapping software interface (e.g., limit total number of links, each newly created link points by 
default from left to right, position by default final outcome nodes at right most portion of screen) - constraints that 
can be implemented in future versions of our mapping software called jMAP (Jeong, 2010b), specifically developed 
and used for this case study. To address these discussions, this case study examined two research questions: 

1 . Which structural attributes (total links, temporal flow, outcome node location) are correlated with accuracy? 

2. What is the relative magnitude of each attribute’s impact on accuracy? 
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Method 



Participants 

Nineteen graduate students (8 male, 1 1 female) enrolled in a Masters level online course on computer- 
supported collaborative learning at a large southeastern university participated in this study. The participants ranged 
from 22 to 55 years in age, and the majority of the participants were enrolled in a Master’s level program in 
instructional systems/design. 

Procedures 

The course examined factors that influence collaborative learning and instructional strategies associated with 
each factor. In week 2 of the course, students used a Wiki webpage to share and construct a running list of factors 
believed to influence the level of learning or performance achieved in group assignments. Students classified and 
merged the proposed factors, discussed the merits of each factor, and submitted votes on the factors believed to exert 
the largest influence on the outcomes of a group assignment. The votes were used to select a final list of 14 factors 
that students individually organized into causal diagrams. 

In week 3, students were presented six example diagrams to illustrate the characteristics and functions of causal 
diagrams. Students were then provided a MS Excel-based software program called jMAP (pre-loaded by the 
instructor with nodes for each of the 14 selected factors) to construct their first causal map (see Figure 1). The 
purpose of each student’s map was to graphically explain their understanding of how the selected factors influence 
learning in collaborative settings. Using the tools in jMAP, students connected the factors with causal links by: (a) 
creating each link with varying densities to reflect the perceived strength of the link (1 = weak, 2 = moderate, 3 = 
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strong); and (b) selecting different types of links to reveal the level of evidentiary support (from past personal 
experiences) for the link. The course instructor also used jMAP to construct an expert map that was used in this 
study to assess the accuracy of students’ maps (see Figure 2). Students were permitted to omit any factors that 
he/she did not believe to directly or indirectly influence the learning outcome. Personal diagrams were completed 
and electronically uploaded within a one -week period to receive class participation points (class participation 
accounted for 25% of the course grade). The diagrams were also used to complete a written assignment describing 
one’s personal theory of collaborative learning (due week 4, and accounting for 10% of course grade). 

Once all the students submitted their first causal map, the instructor used jMAP to download and aggregate 
all diagrams (n = 17) to produce and share with students a matrix conveying the percentage of diagrams that 
possessed each causal link. For example, the matrix in Figure 3 shows that the causal link between ‘Individual 
Accountability’ and ‘Learner Motivation’ was observed in 47% of students’ diagrams. The links highlighted in 
yellow in the matrix above (on the right) identifies the common links observed in 20% or more of the students’ 
diagrams (note: this criterion was specified by the instructor when aggregating diagrams in jMAP). Presented in the 
left matrix are the mean strength values of only those links observed in 20% or more of the diagrams. The 
highlighted values reveal links that are present or absent in the expert’s map (i.e., dark green = links and strength 
values match, light green = links match, but strength values do not, gray = missing target links). 

In week 9, students were presented the matrix revealing the percentage of diagrams (map 1 ) that possessed 
each link. Students posted messages in online threaded discussions to explain the rationale per link. Each posted 
explanation was labeled by students with the tag ‘EXPL’ in message subject headings. Postings that questioned or 
challenged explanations were tagged with ‘BUT.’ Postings that provided additional support were tagged with 
‘SUPPORT.’ In weeks 9 and 10, students searched and reported quantitative findings from empirical research in a 
Wiki to determine the instructional impact of each factor. 

Finally, in week 10, students reviewed the discussions from week 9. Within each discussion thread for each 
examined link, students posted messages to report whether they rejected or accepted the link (along with 
explanations). At the end of week 10, students revised and submitted their causal maps (map 2) based on their 
analysis of the arguments presented in class discussions. 



Measurement 

The number of total links was measured by counting all links in each student’s diagram. Temporal flow 
was computed by dividing the number of right pointing links to the number of left pointing links. Links that were 
perfectly aligned in a vertical position (pointing straight up or straight down) were not included in the computation. 
Position of outcome node was based on the number of pixels separating the left edge of the screen to the left edge of 
the outcome node. By using the matrices automatically generated by jMAP (see Figure 4) to identify the causal root 
links and root cause nodes shared between each student’s map and the instructor’s map, the ratio of correct/incorrect 
root nodes and the number of correct root links were computed. 

Analysis 



The study applied the linear regression via SPSS 17.0. The diagrams produced before and after discussions 
were analyzed by using two regression models: 

Model 1: Ratio of correct root causesi=p 0 + Pi(number of total link;) + p 2 (ratio of temp flow;) + p 3 (outcome 
node location) 

Model 2: Number of correct root links;=p 0 + Pi (number of total link;) + p 2 (ratio of temp flow;) + p 3 
(outcome node location) 
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Figure 4. Matrix representation of student 8’s causal map (causes listed by row, effects listed by column with green 
cells representing causal links correctly identified and blank columns indentifying root causes. Student explanations 
for causal links are stored in comments that can be accessed by placing the mouse over the red triangles. 



Results 



Correlations between attributes and accuracy 

In the student’s initial maps (map 1) produced prior to discussion, temporal flow was negatively correlated 
to the number of correct root links (r=-.461,/>=. 047), and outcome node position was negatively correlated to the 
number of correct root links (r=-.465, jo=.045). In the maps produced following discussion (map 2), temporal flow 
was positively correlated with ratio of correctly/incorrectly root causes (r=.688, />=.003), while total causal links was 
negatively correlated with number of correct causal root links (r=-,523, j9=.037). 

Relative magnitude of attribute impact 

The regression model for the ratio of correct/incorrect root causes following online discussions was found 
to be statistically significant (F(3,12)=5.025, p=.017). The model explains 44.6 % of the variance (Adjusted 
R 2 =.446) and power was 0.73. In this model, temporal flow was the most highly and positively correlated to the ratio 
of correct/incorrect root causes ((3=.772, p=.004), while total causal links showed relatively stronger negative 
correlation than the outcome node position. 
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Regardless of model significance, the regression model for the number of correct root links following 
online discussion explains 28.1% of the variance (Adjusted R 2 =.281). In this model, total causal links was the most 
highly and negatively correlated to the number of correct root links (P=-.554, p=.028), while temporal flow showed 
relatively stronger positive correlation than outcome node position. 



Table 1 Correlations between variables 



Variables 


TL 


TF 


NP 


RC 


RL 


Prior to online discussion 












Total causal links (TL) 


1 










Ratio of temporal flow (TF) 


.028 


1 








Outcome node position (NP) 


.334 


.254 


1 






ratio of correct/incorrect root causes (RC) 


-.213 


-.432 


-.381 


1 




number of correct root links (RL) 
Following online discussion 


-.165 


-.461* 


-.465* 


.541* 


1 


Total causal links (TL) 


1 










Ratio of temporal flow (TF) 


.023 


1 








Outcome node position (NP) 


.159 


.303 


1 






ratio of correct/incorrect root causes (RC) 


-.261 


.688* 


.085 


1 




number of correct root links (RL) 


-.523* 


.352 


.153 


.492 


1 


*p<.05. **p<.001. 












Table 2 The unstandardized and standardized regression coefficients for the variables 






Variables ratio of correct/incorrect root causes 


number of correct root links 


B 


SE 


P 


B 


SE 


P 


Prior to online discussion 












Total causal links -.448 


1.606 


-.065 


-.015 


.037 


-.089 


Ratio of temporal flow -.805 


.508 


-.362 


-.020 


.012 


-.372 


Outcome node position -.040 

Following online discussion 


.037 


-.267 


-.001 


.001 


-.341 


Total causal links -2.935 


2.176 


-.263 


-.247 


.099 


-.554* 


Ratio of temporal flow .796 


.223 


.722** 


.014 


.010 


.321 


Outcome node position -.021 


.047 


-.092 


.001 


.002 


.143 



*p<.05. **p< 001. 



Discussion 

The findings in this case study (though not conclusive) suggests that asking students to position nodes in 
temporal sequence might inhibit students’ ability to identify the correct root causes when students are producing 
their initial causal maps (before discussion). It is possible that when students re-position one node closer to another 
node (but farther away from other nodes) based on the consideration of their temporal relationship, the nodes 
increased distance from other nodes (and reduction in visual proximity) may lead students to skip and omit from 
consideration other possible relationships with a given node. In other words, imposing temporal flow early in the 
map construction process may inhibit the brainstorming process and consideration of all possible relationships 
between nodes. As a result, this might push students to prematurely take specific courses of actions that lead to less 
accurate maps. 

However, the findings also suggest that once students are given the opportunity to discuss and compare 
their maps (and have winnowed down in number the possible cause-effect relationships), imposing temporal 
sequence may actually help students correctly identify root causes. The results show that an increase in temporal 
flow by one standard deviation while holding total causal links and outcome node location constant can potentially 
increase the ratio of correct/incorrect root causes by .722 standard deviations. One possible explanation for this 
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finding is that the process of positioning nodes in temporal sequence creates new options or opportunities to 
articulate and refine the causal chains and identify the causes that mediate root causes and outcomes. This finding is 
somewhat contrary to previous findings where hierarchical structure was found to have no effect on accuracy (Ruiz- 
Primo, 1997). However, computing temporal flow in each student’s map in this particular case study did not pose 
any methodological problems as it did in the study conducted by Ruiz-Primo at al. The differences in measures 
between this study and the study by Ruiz-Primo at al. may have contributed to the differences in findings. 

The other main finding in this study suggests that if a limit is imposed on the number of causal link within a 
map to promote parsimony (or if students are encouraged to reduce the number of causal links in their maps), 
students are better able to correctly identify root cause links. This finding was consistent with the negative 
correlation found between total links and ratio of correct/incorrect root causes. A plausible explanation for this 
finding is that the students that tended to insert excessive numbers of links into their maps may have been the 
students that: (a) tended to link all nodes that are causally related regardless of whether they are directly or indirect 
related; and (b) are not able to identify the correct causal chains and mechanisms underlying the complex 
phenomenon/problem. 



Future Research & Development 

The findings in this study are not conclusive. Nevertheless, the preliminary findings provide ideas as to 
what and when to impose specific types of tasks and/or software constraints on the causal mapping process. Some 
directions for further research are the following: a) control for individual differences in knowledge and 
understanding of the concept/problem under study in order to fully determine the effects of students’ knowledge and 
skills with causal maps on map accuracy; b) increase size of sample and data corpus; c) set the default location of 
the outcome node at the center of the screen rather than to the right portion of the screen in order to fully assess the 
effects of initial node location; d) measure final node location relative to the right edge of the screen (rather than left 
edge) if temporal flow if left to right rather than right to left; e) integrate these rules/constraints into jMAP to 
conduct a controlled experimental study to test and determine the effects of limiting number of links, manipulating 
the option to create links that can point in any or in only one direction, and intentionally varying the default location 
of outcome nodes; f) consider how the effects of each constraint vary when examining causal maps across different 
domains or topics that are or are not naturally temporal in nature; and g) test other metrics for assessing the accuracy 
of students’ maps in relation to an expert map or in relation to a map representing that of the collective group. 
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Abstract 

The purpose of this study was to examine the effects of scaffolding in a form of inquiry questions on 
students’ critical thinking and satisfaction in online argumentation. Twenty-two graduate students in a distance 
learning course were randomly assigned to one of two groups. Each student in the experimental group participated 
in an online discussion activity a) with guided inquiry questions or b) without these questions. Results indicated that 
although providing guided inquiry questions had a positive effect on students’ critical thinking, it did not have an 
effect on students’ satisfaction. These findings and previous research in this area imply that scaffolding such as 
guided inquiry questions can be used to promote students’ critical thinking in online learning environments and 
increase students’ positive attitudes toward online learning. 

Introduction 

In online settings, it is important that instructors provide scaffolding to enhance students’ critical thinking 
skills and maintain learner motivation, especially when students are involved in a task that requires higher order 
processes such as critical thinking. Online argumentation is widely used as a strategy enhancing students’ critical 
thinking process (Nussbaum & Schraw, 2006) and facilitating students’ understanding of content. In most cases, 
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however, online students are asked to communicate with learning materials alone. There are many challenges 
confronting instructors in online discussion settings, such as lack of efficiency of conversation, interactive guidance, 
and immediacy of feedback (Wang & Woo, 2007). Without well structured guidance and instructional supports, 
students may therefore not achieve their learning goals and lose learning motivation. 

Providing guided inquiry questions may supports students’ critical thinking skills in learning contexts 
where the instructor’s involvement is limited (King, 1992; Belland et al., 2008; Gillies & Khan, 2009). Guided 
inquiry questions are a set of questions that require students to focus on key factors in learning materials. These 
questions help students gather information and construct evidence-based arguments. Several studies have 
investigated the effect of using the inquiry questions in online discussion and argumentation (e.g., Bradley et al., 
2008; Ge, Chen, & David, 2005; Golanics & Nussbaum, 2008; Oh & Jonassen, 2007). The Ge, Chen, and David 
(2005) study showed that the elaborated features in inquiry questions helped students to consider, compare, and 
determine a reasonable solution as well as to evaluate those thinking processes in ill-structured complex contexts. 
Furthermore, providing these questions also may increase students’ satisfaction, because guided inquiry questions 
may decrease students’ cognitive load which in turn increases learner achievement. Palmer and Holt (2008) found 
that providing instructional supports in online courses has a positive effect on students’ satisfaction. 

Although recent studies have attempted to show the effect of the questioning technique in online 
argumentation reflecting a complex problem (Cho & Jonassen, 2002), few empirical studies provided content- 
oriented inquiry questions. Earlier studies mostly provided procedural-oriented inquiry questions which only provide 
directions on how to participate in online argumentation. In addition, no studies have assessed students’ critical 
thinking skills through a separate post-test based on a case analysis problem. Furthermore, few studies have 
investigated students’ satisfaction with online learning activities using guided inquiry questions. 

The purpose of this study was to examine the effects of guided inquiry questions on students’ critical 
thinking skills and satisfaction in online argumentation. One group of students in our study received guided inquiry 
questions while they participated in an online discussion; the other group did not receive these questions. Students’ 
critical thinking skills were measured by an essay writing test. This study defined the critical thinking skills as 
students’ ability to gather and analyze relevant information in order to draw evidence -based conclusions and 
generate solutions. In addition, the study evaluated students’ satisfaction with their online learning activities. 

The study hypothesized two results. First, students who were provided guided inquiry questions would 
score significantly higher on the posttest than their counterparts. Existing research has found that inquiry questions 
promote critical thinking in online argumentation (Zydney, 2008; Ge & Land, 2003). Second, students who 
participated in online argumentation with guided inquiry questions would indicate higher satisfaction than students 
who participated without inquiry questions. Without these questions, students should expend more cognitive effort, 
which may cause cognitive load and decrease students’ motivation. 



Method 



Participants 

The participants in this study were 22 graduate students enrolled in an online graduate course at one of the 
state universities in Florida. Five students were attending the doctoral program and 17 students were attending the 
masters program in the College of Education. Nine students were male and 13 students were female. There were 3 
international students (all from East Asia) and 19 Americans. All students gave their consent to participate in this 
experimental study. One student did not participate in the manipulated activity and one student did not submit the 
post-test. Since this study regarded those two participants as missing data, there were a total of 20 data sets. 

Materials 

This online course was served by its own website embedded in BlackBoard which is widely used as a 
learning management system in many universities in the US. The course website serves text-based weekly 
instruction including learning objectives, hyperlink resources, and activity descriptions. During week 7, students 
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were required to read book chapters and journal articles about needs and barriers in developing distance education. 
All students had opportunities to review what they read in a summary page which contained important information 
excerpted from the week 7 readings. The information in the summary page guided students to consider key factors in 
developing a distance education program. To make sure students understood all the key factors, an optional quiz 
was provided. In the subsequent online argumentation, the case analysis problem required that students argue 
whether or not a make-believe country should develop a distance education program. Online discussion forums were 
developed by functions in BlackBoard system. 

Independent Variable 

The independent variable in this experiment was the set of guided inquiry questions for the online 
argumentation. The guided inquiry questions, constructed by synthesizing given reading materials, were designed to 
guide students in applying important information to a situated learning activity. Students in the treatment group 
participated in online argumentation with the guided inquiry questions, while students in the control group 
participated in online argumentation without the guided inquiry questions. 

The set of guided inquiry questions (totaling 19 questions) was developed by an instructor and three 
graduate assistants. The questions covered nine key factors and one supplementary factor, so that students could 
justify their decision-making on the case analysis problem by responding to those questions. Given factors in 
content-oriented inquiry questions were matched with the information presented in the summary page. The questions 
were expected to assist students in elaborating rationales for supporting their arguments. For example, having a 
question such as “what kind of media may support the delivery of a distance program?” might lead students to 
reason whether the make-believe country has an appropriate delivery condition for distance education or not. A set 
of inquiry questions was provided on the left side of the screen as a separated frame (see Figure 1), so that students 
could follow the questions without changing the screen while analyzing the given case and writing evidence-based 
arguments. 
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Figure 1. Screen capture of the BlackBoard discussion forum with guided inquiry questions 

Dependent Variables 

This experimental study measured students’ critical thinking skills and students’ satisfaction after finishing 
the required online argumentation activity. 
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Students’ critical thinking skill was measured by a post-test which asked students to write an essay of at 
least 300 words. In the essay test, students were given a new case analysis problem in which they were asked to 
determine whether a different make-believe country should develop a distance education program or not. The test 
instructions indicated that students should provide a rationale for their position. To evaluate the essay, the instructor 
and three graduate assistants developed a standard assessment rubric based on McLean’s (2005) criteria for 
evaluating the quality of critical thinking: clarity, relevance and breadth (or depth). The standard assessment rubric 
contains 10 items: one item for evaluating clarity in posing the position and rationale; one item for evaluating 
relevance of focus on the topic and the case; and eight items for evaluating how broadly or deeply the evidence is 
considered in their reasoning process. To ensure a fair assessment, required elements in the last eight items were 
matched with the key factors in the summary page, which every student could access. The score range for each item 
was 0 to 2 points. The possible maximum score was 20 points. 

The students’ satisfaction survey was conducted by an online assessment tool in BlackBoard, using 10 
items. Seven items focused on learner-content interaction and three items on general satisfaction, based on 
Strachota’s (2006) five constructs of measuring student satisfaction with online learning. Learner -content 
interaction is satisfaction with the subject matter which includes course content, lessons, learning activities, 
assignments, and the course website. General satisfaction focuses on whether students’ needs have been met 
(Strachota, 2006). Responses to the items were in the form of a five-point Likert scale: strongly agree, agree, neither 
agree nor disagree, disagree, and strongly disagree. 

Procedures 

Stratified random assignment was used to allocate subjects to the treatment group with guided inquiry 
questions and the control group. Because of the small sample size, pure random assignment potentially threatened 
the homogeneity between the two groups. In this experiment, gender and level of participation were used in the 
stratified random assignment. In particular, the number of messages posted in the previous online discussion was 
used to determine the stratification of high and low participation. Each group had 1 1 students. After eliminating the 
missing data, the number of participant in each group was 10 subjects. 

During week 7, students in both groups were asked to finish given readings and to review the readings 
through the summary page and the optional quiz. They were then required to participate in online argumentation 
with the same case analysis problem, but using two different web pages. Students could access only one of two 
group discussion forums. The activity description indicated that students in both groups should write their rationale 
to support their determined position by posting a minimum of 4 messages in the discussion forum. In the online 
argumentation, only those students working with the guided inquiry questions were asked to answer each question to 
support their arguments, while students in the control group made their arguments without guided questions. 

During week 8 and week 9, as a portion of the midterm exam, students were required to write an essay of at 
least 300 words about the new case analysis problem. Two graduate assistants evaluated the students’ essay writings 
based on the standard assessment rubric. In addition, students were asked to complete the online satisfaction survey. 

Results 

All statistical tests were conducted using an alpha level of .05. No serious violation of normality or 
homogeneity of variance was found. The data were analyzed using an independent-sample t test for each dependent 
variable. 

Critical thinking skills 

Two raters separately evaluated the post-test. Both raters used a blind grading method and followed the 
standard assessment rubric for the essay writing. The score range in the standard assessment rubric was 0 to 20 
points. The allowed rate of score difference was 15% (3 points) in this experiment. Based on Pearson’s Correlation, 
the inter-rater reliability for the post-test was .876. A review of the data revealed that there was no violated 
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assumption of normality and homogeneity variance (p=.331 based on Levene’s Test). With alpha set at .05, two- 
tailed condition, and 10 participants per group, the probability of detecting a difference between means was .62. 

Table 1 presents the means and standard deviations on the post-test scores of critical thinking skill across 
the two groups, with and without the guided inquiry questions. T test analysis was applied to compare the mean 
difference between two groups. The group provided with guided inquiry questions scored significantly higher on the 
essay writing test than the group not provided with inquiry questions, i(18) =2.392 , p=. 028. Effect size was <f=1.07, 
indicating a large effect (Cohen, 1988). Consistent with the first hypothesis, students who participated in the online 
argumentation with guided inquiry questions demonstrated higher critical thinking skills than students who 
participated without inquiry questions in the post-test. 

Table 1 



Means and standard deviations of students' critical thinking skill across groups 



Groups 


Essay writing test a 






Mean 


SD 


Without guided inquiry questions 
(n=10) 


13.40 


2.97 


With guided inquiry questions 
(n=10) 


16.35 


2.53 



a Maximum possible score was 20. 



Satisfaction 

Of the 10 items in the students’ satisfaction survey, the first 3 items measured general satisfaction and the 
last 7 items measured satisfaction with the learner-content interaction. Each item was answered using a five-point 
Likert scale. Based on Cronbach’s Alpha, satisfaction survey items had a reliability of .894. 

Table 2 presents the means and standard deviations for the satisfaction survey for the two groups. T test 
analysis was applied to compare the mean difference between two groups. The groups did not differ significantly on 
general satisfaction, 1(18) =.109, p=.362, on learner-content interaction, 1(18) =.113 , p=.635, or on total satisfaction, 
1(18) =.116 , p=.467. Students who participated in online argumentation without guided inquiry questions expressed 
a level of satisfaction with their learning activities as high as students who participated in online argumentation with 
guided inquiry questions. 

Table 2 



Means and standard deviations of students' satisfaction across groups 



Groups 






Satisfaction survey a 








General satisfaction 


Learner-content 

interaction 


Total 






Mean 


SD 


Mean SD 


Mean 


SD 


Without guided inquiry questions 
(n=10) 


4.17 


.63 


4.16 .53 


4.16 


.54 


With guided inquiry questions 
(n=10) 


4.20 


.76 


4.19 .58 


4.19 


.62 



Note. Among total 10 items in satisfaction survey, iteml through item 3 asked general satisfaction and item 4 
through item 10 asked satisfaction of learner-content interaction. 
a A five-point Likert scale ranged from 1 (strongly disagree), to 5 (strongly agree). 



Discussion 

Supporting the first hypothesis of the study, students provided with guided questions showed a higher level 
of critical thinking in their evidence-based arguments than students not provided with these questions. The positive 
effects of guided questions on critical thinking were believed to be because students were able to use various points 
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of view. This finding is consistent with past studies which have shown that guided questions stimulate students’ 
deep thinking process to generate evidence-based arguments in solving a complex problem (Bradley et al., 2008; Ge, 
Chen, & David, 2005; Golanics & Nussbaum, 2008; Oh & Jonassen, 2007). 

This positive effect of the guided inquiry questions can be explained by the fact that responding to inquiry 
questions helps students’ cognitive and metacognitive activities. The process of responding to inquiry questions 
stimulates students to analyze information in a given complex context, and then thinking more broadly or deeply to 
articulate, justify, and reflect on their arguments (Ge & Land, 2003). In this study, the students with guided inquiry 
questions were well scaffolded to foster their critical thinking skills which involved processes of analysis, 
articulation, justification, and reflection. 

Contrary to the second hypothesis, there was no difference between two groups in relation to levels of 
satisfaction. Students in both groups reported high general satisfaction and high satisfaction of learner-content 
interaction. There may not be one reason for their high level of overall satisfaction, because different learners have 
different perceptions or standards in relation to their learning. In the satisfaction survey, some of descriptive 
responses said that the argumentation activity motivated students in the online course. The online argumentation has 
been considered an alternative strategy to engage students more in online discussion (Golanics & Nussbaum, 2008). 
In addition, some students mentioned that they felt an authenticity from the invented case which presented a 
complex problem that was likely to happen in the real world. 

In distance learning, improving students’ critical thinking and increasing students’ motivation are important 
issues. These findings provide empirical evidence for distance learning educators and instructional designers to use 
guided inquiry questions when they want to enhance students’ critical thinking as students solve complex problems 
in online learning environments. Although this study did not support the effect of scaffolding on students’ 
satisfaction with online learning, providing appropriate scaffolding may increase students’ motivation in terms of 
confidence and satisfaction. This study indicates that providing guided inquiry questions may be a good strategy to 
improve students’ critical thinking skills and suggests future research on the effects of scaffolding on students’ 
satisfaction in online learning would be valuable. 

Limitations and future suggestions 

The study had some limitations. First, we administered the post-test immediately after the one week online 
argumentation activity. Consequently the results do not indicate the effect of the guided inquiry question on 
retention of improved critical thinking skills. Second, the post-test provided a new case analysis problem which 
was very similar to the case provided in the online argumentation. As a result we cannot assume that students can 
transfer their critical thinking skills to new learning situations. Additional research is needed in which a long-term 
assessment with different cases analysis problems in the post-test is administered. 

In future research, the attributes of participants could be treated as an independent variable in order to 
determine how the guided inquiry questions might affect learners who differ by gender, age, prior experience in 
either domain knowledge or an online course, and the level of self-regulation. Moreover, there are other types of 
scaffolding suggested by other researchers in the area of pedagogical approaches (Hmelo-Silver et al., 2007). In the 
satisfaction survey one student described the need for a step-by-step exercise in analyzing the given case which can 
be presented as a fill-in response by sequential questions. Building on this, future research could consider the 
different types of scaffolding such as the process work sheet and the worked example in evidence-based 
argumentation. 
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Abstract 

Is there any generational difference among Hispanic learners in the arena of e-learning? This 
quantitative inquiry is intended to study students’ technology ability, learning activity preference, 
attitude towards technology use, and instructional strategy orientation at a Hispanic serving 
institution of higher education in a university along the southern border with Mexico. The finding 
from a previous study used a straightforward t-test comparison of variance and determined that 
age was the dominant factor affecting learners’ orientation toward instructional strategy. However, 
does this observation still hold true after controlling for the students' preferred course delivery 
mode and academic status? Limitations and recommendations are also be stated. 

Introduction 

Past literature of generational differences seems to acknowledge the significance of attending to unique 
characteristics of each age generation (Aro, Rinne, Lahti, & Olkinuora, 2005; Borrero, McGinnis, McNeil, Frank, & 
Conigliaro, 2008; Hammill, 2005; Mishler & Rose, 2007; Waddell, 2004). 

This quantitative inquiry was designed to investigate generational differences in the (a) learners’ 
technology ability (TA), (b) learning (activity) preference (LP), (c) attitude toward use of technology (e.g., email) 
that are commonly used in learning online (ATT), and (d) instructional strategy orientation (IS) within a Hispanic - 
serving institution on the southern border with Mexico. More importantly, the investigation is intended to synthesize 
findings and extrapolate implications for practitioners and researchers interested in a similar study. 

Besides the four latent factors previously mentioned, Hispanic learners’ demographic information (e.g., age) 
is taken into account in this research. Generational differences in this quantitative study may refer to any cognitive, 
psycho-motor, affective, and interpersonal discrepancies of learning online due to age differences. These 
differences among generations (i.e., Baby Boomers, Generation X, and Net Gen) appear to raise an instructional 
design issue for professors who teach online and instructional developers who assist the faculty in developing online 
courses (Feiertag & Berge, 2008). The issue can be multi-faceted. Two of these issues which the authors hoped to 
address can be understood as follows. 

1 . Are there any differences between age groups in their respective perceptions or beliefs of online learning? 
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2. To what extent do these differences in the four factors affect the decision-making process of the course 
design and development? 

The results of this survey study were expected to inform online course developers and cyber instructors of potential 
instructional strategies that may be taken into account in the design and development of a course with an online 
component integrated. It was also anticipated that their students will better enjoy their learning experience in an 
online environment in light of this study. 



Literature Review 

Using language acquisition as an analogy, Prensky (2001) used two terms, “digital natives” and “digital 
immigrants” in an attempt to explain the contrast between those who grew up with the Internet and digital devices 
and those who were born before the Internet was created. Native speakers acquire their first language or mother 
tongue effortlessly compared to those who speak the language as a second or foreign language. In this context, 
digital natives are characterized as those who are relatively comfortable or proficient in terms of interacting with the 
computer and other technology devices. Such comfort or proficiency is so pronounced that it can be associated with 
speaking a native language. Digital immigrants are those who did not grow up with the Internet, so the way they 
interact with the computer or the Internet may not be similar, possibly not even as comfortable, as their generally 
younger counterparts. Furthermore, Prensky (2005) claimed that younger learners are now able to achieve learning 
through a mobile phone. Older learners would not normally consider such an activity or behavior to be a typical, 
common place behavior for learning in everyday life other than possibly calling “411” for information. Despite that 
fact that empirical findings seemed missing from Prensky’ s arguments, this noticeable dichotomy has suggested that 
as more and more digital natives are going to college, a study on whether or not college courses ought to be 
redesigned in an attempt to address the dichotomy is not only timely, but has gained further traction (Carlson, 2005). 

Are digital natives and digital immigrants so different? There seemed to be a generational difference in 
learning soft skills (a.k.a. people skills) across “Traditionals,” “Baby Boomers,” “Generation X,” and “Generation 
Y” (Tolbize, 2008). Tolbize (2008) found that younger workers prefer assessment and feedback whereas older 
generations favor classroom instruction. Even so, strategies such as “on the job training, discussion groups, peer 
interaction and feedback, and one-on-one coaching” are popular across the four generation groups. On the other 
hand, Tolbize contended that when it was concerned about hard or technical skills training these workers prefer a 
similar instructional design. This is one observation from the workforce. Another observation came from a report by 
Reeves (2008). Reeves stated, “Generational differences are evident in the workplace, but they are not salient 
enough to warrant the specification of different instructional designs or the use of different learning technologies” (p. 
21). 

What about higher education? According to Garcia and Qin’s (2007) findings, the Net Generation (i.e., 
digital natives) and non-Net Generation (i.e., digital immigrants) may differ from each other with regard to the 
ability to operate a technological device in favor of the natives. However, both groups showed no difference in their 
perception of an effective instructional activity; both desired more interaction with the instructor through class 
lectures and discussions. Stapleton, Wen, Starrett, and Kilburn (2007) found that Millennial (i.e., Net Gen) have a 
tendency to interact or collaborate with their peers through various technologies, but when it comes to their own 
perceptions of learning in online courses, there is not any difference than other age groups. The four researchers 
warned that the age differences should not be considered the only single factor that determines students’ successful 
learning or satisfaction with online learning. Billings, Skiba, and Connors (2005) compared the undergraduate 
nursing students’ (i.e., mostly Net Gen) perception of experiences in Web-based courses to their graduate nursing 
students (i.e., primarily Generation X). As a result, they discovered that Net Gen likes to take the responsibility for 
their own learning and develops a similar perception for the use of technology and a similar concept of 
professionalism, just like other generations, regardless of some differences in educational practices. Thus far, the 
research on generational differences has not clearly shown much significant difference between Net Gen and non- 
Net Gen with respect to a favored learning environment. This is one of the motivations behind this investigation. 
Since most of the studies mentioned earlier targeted primarily non-Hispanic student population, another motivation 
of this paper lies in a question, “How do the two age groups differ in the e-learning environment that predominantly 
services a Hispanic student body?” 

This cursory review of the relevant literature suggested that, at present, there is a potential gap in the 
literature regarding the possible effect of ethnicity, possibly compounded by age in students' effective participation 
in online learning. The present investigation was intended to explore this generational differences issue by targeting 
Hispanic learners, specifically Mexican American learners. One primary question investigated in this research effort 
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reads as follows, “Is there any significant difference between age groups in the four factors (i.e., TA, LP, ATT, and 
LS)?” Three follow-up questions were attempted to further the inquiry related to e-learning. 

1 . Is there any significant difference among three course delivery modes in the four factors? 

2. Is there any significant difference between academic status groups (graduate vs. undergraduate) in the four 
factors? 

3. Does the effect of age on the four factors remain significant when controlling for course delivery mode and 
academic status? 

Method 

As aforementioned, this quantitative study was designed to study how age can affect TA, LP, ATT, and LS 
using a Hispanic student population. The setting of this quantitative study was located in the online learning 
environment, empowered by Blackboard Course Management System within a state university in South Texas, 
U.S.A. According to Santiago (2006), the university is a Hispanic-serving institution (HIS) with approximately 94 
percent of total undergraduate Hispanic full-time equivalent student enrollment. 

Target population was the entire student body, including both graduate and undergraduate, that is enrolled 
in any class that has an online Blackboard component in it. The accessible population consisted of those students 
whom investigators are given instructors' permission to survey. The surveyed students must be at least 1 8 years old 
and participate in this study on a voluntary and anonymous basis. As suggested by Hartman, Moskal, Dziuban 
(2005), participants are divided into three age groups: (a) those born before 1965 (aka Baby Boomers), (b) those 
born in 1965 through 1980 (aka Generation X), and (c) those born after 1980 (aka Net Gen). 

This study was launched in the summer of 2008. As of September 2009, 1 77 students have successfully 
responded to the questionnaire. Of all, 140 are Hispanic learners, with 1 1 identified as baby boomers, 54 as of 
Generation X, and 75 of Net Gen. Additionally, two out of 140 cases were removed from the dataset because of 
missing data. 

Data were collected using a password-protected computer server through an online questionnaire. The 
questionnaire was composed of six instruments, but for the purpose of this phase of investigation. Behavior Pattern 
Instmment was excluded. The five instruments are as follows: 

1 . Demographics Instalment (9 questions), measured on a nominal scale 

2. Technology Ability Instalment (25 questions), measured on an interval scale 

3. Learning Preference Instalment (27 questions), adapted from the work by Loo (2004), measured on an 
interval scale 

4. Attitude Instrument (20 questions), adapted from the work by Ajzen and Fishbein (1980), measured on an 
interval scale 

5. Instaictional Strategy Instalment (8 questions), adopted from the work by Tapscott (1997), measured on an 
interval scale 

Demographics Instrument dealt with variables, such as sex, birth year, and academic major. Technology Ability 
Instalment included questions that deal with respondents’ perceived ability in technology use, such as “I can use 
Google Docs Spreadsheet” and “I can download photos from a digital camera.” The Learning Preference Instrument 
was concerned with respondents’ preference in the classroom activity. Of all, two questions were, “I like in-class 
group projects as a class activity” and “1 like writing reflection as a class activity.” The Attitude Instrument was 
composed of questions in regard to respondents’ attitude toward technology use. Two of the questions/statements 
included were, “All things considered, my use of Blackboard is negative vs. positive” and “All things considered, 
my use of electronic devices is negative vs. positive.” Instaictional Strategy Instrument encompassed questions with 
respect to respondents’ preference of strategies commonly used in the instaictional setting. Two sample 
questions/statements from that instalment were, “All things considered, my preference of instructional strategies 
below is teacher-centered vs. learner-centered” and “All things considered, my preference of instaictional strategies 
below is school as torture vs. school as fun.” 

Technology Ability and Learning Preference Instalments both were measured on a five-point Likert scale 
with “1” as strongly disagree, “5” as strongly agree, and “3” as neutral. The higher the composite scores are on each 
of the two instalments, the more agreeable the respondents are. On the other hand, Attitude and Instructional 
Strategy Instalments both adopted a five-point bipolar semantics scale. The higher the composite scores are Attitude 
Instalment, the more favorable or positive the responses are. For Instaictional Strategy Instalment, the higher scores, 
the more learner-centered. The questionnaire was administered at one occasion in the beginning of the semester, 
which begins in the third week and end in the fifth week of the Fall and Spring semesters. In both Summer semesters 
in 2008 and 2009, the administration began in the second week and end in the fourth week. With instructors' prior 
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approval, investigators emailed the consent form through Blackboard email system. Data sets will be purged from 
all the computer hard drives in five years after the research begins (by the end of spring 2013) as regulated by the 
institutional review board that approved the project in May 2008. 

Data sets were exported from the password-protected computer server to Microsoft Office Excel before 
entered to the SPSS program. Due to the number of the participants, the baby boomers group and the Generation X 
group were combined. The merged group was then named, Non Net Gen. Among others, statistical procedures, such 
as multivariate analysis of variance (MANOVA) and multivariate analysis of covariance (MANCOVA), were 
adopted for further data analysis. 



Results 

A Cronbach’s alpha reliability testing was conducted to test the reliability of the four instruments on the 
interval scale adopted or adapted from the literature. Alpha values were .92 for Technology Ability (TA) 

Instillment, .85 for Learning Preference (LP) Instrument, .93 for Attitude (ATT) Instrument, and .77 for 
Instructional Strategy (IS) Instrument. The results indicated that the four instmments were reliable and all were 
consistent with earlier studies with fewer participating students (e.g., Pan, Zhang, & Sullivan, 2009). In replicating 
the earlier phase of the investigation using a slightly larger sample and a different statistical procedure, MANOVA 
was used, Wilks’s Lambda = .88, F( 4, 125) = 4.46, p =.002. With ANOVA as a follow-up procedure and tested at 
the .0125 level, of all, the scores on ATT and IS were found significant, F( 1 , 128) = 11.66 ,/? = .001 and,F(l, 128) = 
9.61, p = .002. 

To answer Research Question 1, “Is there any significant difference among three course delivery modes in 
the four factors?” MANOVA was used, Wilks’s Lambda = .82, F(S, 222) = 4.46 , p =.004. With ANOVA as a 
follow-up procedure and tested at the .0167 level, of all, the scores on IS were found significant, F( 2, 114) = 6.65, p 
= . 002 . 

To answer Research Question 2, “Is there any significant difference between academic status groups 
(graduate vs. undergraduate) in the four factors?” MANOVA was used, Wilks’s Lambda = .79, F( 4, 103) = 4.46. p 
<.001. With ANOVA as a follow-up procedure and tested at the .0125 level, of all, the scores on IS were found 
significant, F( 1, 106) = 11.42,/? = .001. 

To answer Research Question 3, “Does the effect of age on the four factors remain significant when 
controlling for course delivery mode and academic status?” A MANCOVA result indicated that the effects of age on 
the four factors do not remain significant when controlling for course delivery mode and academic status, Wilks’s 
Lambda = . 94, ,F(4, 123) = 2.04,/? = 093. 



Conclusions 

The primary purpose of this phase of investigation with a larger sample (V=138) was to study the effect of 
age on four dependent variables, scores on Technology Ability, Learning Preference, Attitude, and Instructional 
Strategy Instruments using Mexican American students. The MANOVA results indicated that age has a significant 
effect on these students’ attitude toward technology use and their preference of strategies commonly used in the 
instructional setting. Both outcomes were in favor of non-Net Gen group, which is congruent with the earlier 
findings (see Pan, Zhang, & Sullivan, 2009). This suggested that non-Net Gen group has a more favorable attitude 
toward technologies such as Blackboard, email, and electronic devices (in general) than the Net Gen group. This 
also suggested that the non-Net Gen group prefers an instructional strategy that is more non-linear, discovery-type, 
and in line with the learner-centered approach, than the Net Gen group. This finding is not consistent with what 
Prensky (2005) described about the Net Gen group. To further investigate this inconsistency, the authors examined 
any possible effect of two variables, delivery mode and academic status, and their influence each on the four 
dependent variables. 

An attempt to determine whether course delivery mode (i.e. fully Web-based, hybrid, and face-to-face) has 
an effect on the four dependent variables was made using MANOVA. Pos hoc analyses to the ANOVA test for the 
IS scores showed that Fully Web-based group significantly scored higher than the Face-to-face group. Another 
MANOVA result indicated that academic status (i.e., graduate and undergraduate) has a significant effect on 
instructional strategy, suggesting graduate students are more in favor of the learner-centered approach than 
undergraduate students. 

However, when controlling for delivery mode and academic status, age does not seem to exert any 
influence on any of the four dependent variables, suggesting that statistically speaking, the Net Gen and non-Net 
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Gen groups do not differ from each other in Technology Ability, Learning Preference, Attitude, or Instructional 
Strategy Instruments even though the non-Net Gen group seem to outperform the Net Gen group across the four 
variables. This could have been the fact that generally speaking those who participated in this survey study were 
either younger undergraduate students who were taking non-fully Web-based courses or relatively older graduate 
students who were taking fully Web-based courses. Regardless, the findings of this paper echoed previous studies by 
Billings, Skiba, and Connors (2005), Garcia and Qin (2007), Stapleton, Wen, Starrett, and Kilburn (2007). 

So, there may be nuances between the two age groups, but the differences are too subtle to warrant any 
redesign of the online course solely based on the findings of this paper. 

Having said all that above, the findings of this quantitative investigation are by no means expected to apply 
to the entire Hispanic college student body around the country. This was due to its small sample size, the 
volunteered convenience sample, and the accessible population, Mexican American students. It is recommended that 
the sample be enlarged using stratified random sampling for further analysis. 
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Abstract 

Supporting the K-12 teachers involved in adopting a radical educational innovation requires that they be 
understood on a personal level. By capturing teacher concerns at multiple points during implementation, change 
facilitators can design personalized interventions that help educators proceed through the adoption process. A study 
was conducted to explore how the Stages of Concern (SoC) component of the Concerns-Based Adoption Model 
(CBAM) could be used to better understand educators undergoing change and to facilitate their adoption of a mixed- 
reality embodied environment. Eight educators participated over the course of one academic year, with two 
receiving a personalized training intervention. The progress and patterns of adoption amongst the educators were 
systematically tracked and compared to gain insights about teachers experiencing change and how to design support 
interventions to facilitate adoption. In the end, the Stages of Concern proved to be an effective system for tracking, 
understanding, and supporting high school teachers in adopting this technological innovation. 

Introduction 

A personal understanding of each educator involved in the change process is critical to the successful 
implementation of innovative technologies. Change implementation is not an instant in time, but rather is a process 
whose success or failure is determined at the personal level (Hall & Hord, 2005). Understanding individuals is 
crucial when dealing with a radical educational innovation, such as the Situated Multimedia Arts Learning Lab 
(SMALLab). 

SMALLab is a mixed-reality environment that emphasizes embodiment, multimodal feedback, and 
collaboration as vehicles for learning. Its architecture consists of a cube-shaped metal frame with open sides. An 
attached array of motion-capture cameras tracks objects within its boundaries, while a projector and speakers 
provide users with real-time audiovisual feedback (Hatton, Birchfield, & Megowan-Romanowicz, 2008). In this 
system, students simultaneously interact with each other and dynamic multimedia (text, graphics, sounds) through 
three-dimensional movements (Johnson-Glenberg, Birchfield, Megowan-Romanowicz, Tolentino, & Martinez, 
2009). SMALLab has been studied in terms of student learning outcomes and has shown positive results in science 
(Birchfield & Megowan-Romanowicz, 2009, Johnson-Glenberg et al., 2009, Tolentino et ah, 2009) and language 
arts (Hatton et ah, 2008). The system is currently deployed in three U.S. high schools. In the Eastern United States, 
the SMALLab is part of an exploratory curriculum that focuses on game -based learning. In the southwestern United 
States, the newest SMALLab is being used in a student-driven design and development course. Also in the 
southwestern United States, the longest standing SMALLab installation has been used by multiple teachers in 
several subjects and has a history of professional learning communities and research partnerships. For these reasons, 
this study explored the experiences and perceptions of the teachers incorporating SMALLab into their practice at the 
latter high school. 
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The Concerns Based Adoption Model (CBAM) was developed to help facilitators to address individual 
adoption needs (Hall & Hord, 2005). One CBAM component, the Stages of Concern (SoC), describes the 
developmental pattern by which most innovation adopters follow (George, Hall, & Steigelbauer, 2008). Its 
accompanying measure, the Stages of Concern Questionnaire (SoCQ), was used to learn about SMALLab educators. 
Teachers’ responses to the SoCQ addressed the following research questions. 

1 . How can SoC be used to better understand the adopters of a radical educational innovation? 

2. How can SoC be used to design appropriate interventions for the adopters of a radical educational 
innovation? 

3. How can SoC be used to assess the impact of designed interventions? 

Methods 

Participants 

Eight teachers from a high school in the southwestern United States participated in this study (n = 8). This 
location has a dedicated SMALLab installed on-site. The educators have incorporated SMALLab into their 
curriculum to varying degrees. They are diverse in terms of teaching experience and technical knowledge. Four are 
male and four are female. Of the eight teachers, four work in science, three work in English language development, 
and one works in special education. Although these educators received a stipend for participating in SMALLab 
professional learning communities, their decision to partake in this study was voluntary. 

Procedures 

During the fall 2009 semester, between October 22 and November 16, each educator completed the SoCQ 
for the first time. On January 21, 2010, two teachers, who were to receive a training intervention during the spring 
semester, again completed the SoCQ. Near the end of the spring 2010 semester, between March 25 and April 12, 
each of the eight educators completed the final administration of the SoCQ. 

Intervention 

Two teachers, one male and one female, who specialized in English language development, participated in 
a training intervention above and beyond that of the remaining study participants. The individualized training 
intervention was designed to assist the teachers in transforming their roles from primarily end-users of SMALLab 
modules to designers and developers of their own SMALLab content. Training activities consisted of operating the 
SMALLab hardware, generating multimedia and syncing it with the system, and using the SMALLab software to 
create and modify learning scenario templates. The development of these skills had not been previously fostered in 
teachers at the school. Both teachers participated in four two-hour sessions, for a total of eight hours of training. 
Three sessions involved two-on-one instruction (two teachers, one trainer), while the remaining session involved 
one-on-one instruction. The training inteivention took place over a period of five weeks between January 21, 2010 
and February 25, 2010. 

Instmment 

The Stages of Concern Questionnaire (SoCQ) is a 35-item Likert-style survey that can be employed to 
understand the adopters of any innovation (Hall & Hord, 2005). The SoCQ is a well-known and validated instalment 
(Hall, George, & Rutherford, 1977) that has been used, from its inception to today, to understand the concerns of 
innovation adopters (LaRocco & Murdica, 2009, Chamblee, Slough, & Wunsch, 2008, Donovan, Hartley, & 

Stmdler, 2007). It is designed to measure the concerns of individuals as they move through a developmental series 
of seven stages (Figure 1). Hall et al. (1977) described being concerned as "being in a mentally aroused state about 
something" (p. 5). Therefore, concerns can involve both negative and positive emotions and perceptions, as well as 
eveiything in-between. Following the 35 likert-scale items, the SoCQ can be appended with an additional page. In 
this study, open-ended questions regarding the educators' experiences, roles, and concerns in relation to SMALLab 
were included. These items were used to gather further insights about participants by allowing them to provide 
extended descriptions of their perceptions. 
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Refocusing 



Collaboration 



Consequence 



Management 



Personal 



Informational 



Unconcerned 



The individual focuses on exploring ways to reap more universal 
benefits from the innovation, including the possibility of making 
major changes to it or replacing it with a more powerful alternative 



The individual focuses on coordinating and cooperating with others 
regarding use of the innovation. 



The individual focuses on the innovation’s impact on students in his 
or her immediate sphere of influence. Considerations include the 
relevance of the innovation for students; the evaluation of student 
outcomes, including performance and competencies; and the 
changes needed to improve student outcomes. 



The individual focuses on the processes and tasks of using the 
innovation and the best use of information and resources. Issues 
related to efficiency, organizing, managing, and scheduling 
dominate. 



The individual is uncertain about the demands of the innovation, his 
or her adequacy to meet those demands, and/or his or her role with 
the innovation. The individual is analyzing his or her relationship to 
the reward structure of the organization, determining his or her part 
in decision making, and considering potential conflicts with existing 
structures or personal commitment. Concerns also might involve the 
financial or status implications of the program for the individual and 
his or her colleagues. 



The individual indicates a general awareness of the innovation 
and interest in learning more details about it. The individual does 
not seem to be worried about himself or herself in relation to the 
innovation. Any interest is in impersonal, substantive aspects of 
the innovation, such as its general characteristics, effects, and 
requirements for use. 



The individual indicates little concern about or involvement with 
the innovation. 



Figure 1. Stages of Concern About an Innovation. From George et al., 2008. Reprinted with permission. 



Data Analysis 



Educators' responses to the SoCQ were interpreted in adherence to the outlined protocols (George et al., 
2008, Flail et al., 1977). Initially, each person's data were graphed on a diagnostic chart (Figure 2) called a "Stages 
of Concern profile" (George et al., 2008, p. 91). This profile represents the personal concerns of a given individual 
across the seven dimensions at a particular moment in time. A person's profile conveys the relative intensity of his or 
her concerns across all seven stages at that time. In accordance with the recommended interpretation methods 
(George et al., 2008, Flail et al., 1977), the educators' profiles were examined by looking at the overall shape, peaks 
and valleys, and relative position of the stages. Individual SoCQ item responses were also consulted. 
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Results 



For brevity, the results of the two teachers in the intervention group will be described here, along with 
comparisons to the profdes of the nonintervention group. It is hoped that these results will aid others in envisioning 
the information that is derived from SoC research. 

Intervention Group 

Teacher A is a male, high school English language development teacher in his early fifties with several 
years of teaching experience, one year of SMALLab experience, and limited classroom technology integration 
experience. In October 2009, Teacher A's SoC profile (Figure 2) revealed high interest (low stage 0; for this one 
stage only, the scale is reversed) in learning (stage 1) about the SMALLab system and a notable concern for the 
system's impact on students (stage 4). By January 2010, nearly two months removed from SMALLab due to winter 
vacation. Teacher A showed lower interest in SMALLab (stage 0), but sustained a dominate concern in learning 
(stage 1) more about the system. In late March 2010, towards the end of the school year, Teacher A still showed a 
primary interest in exploring the SMALLab (stage 1) and had renewed collaboration concerns (stage 5). His overall 
interest in SMALLab (stage 0) returned to a more moderate level. 




SoC Stages 



Figure 2. Teacher A's SoC profiles in October 2009 (dotted), January 2010 (dashed), and March 2010 (solid). 

Teacher B is a female, high school English language development teacher in her early fifties with a few 
years of teaching experience, no SMALLab experience, and a zest for educational technology. In October 2009, her 



172 





profile (Figure 3) indicated intense interest in SMALLab (low stage 0) and concern about the impact that SMALLab 
would have on her students (stage 4). The remaining intensities in her profile were relatively moderate. By January 
2010, Teacher B's profile was moderate across most stages (0, 2, 3, 4) and her dominant concern was learning more 
about SMALLab (stage 1). In late March 2010, Teacher B's profile exhibited a primary concern over the impact that 
SMALLab has on her students (stage 4). Meanwhile, she had lower early stage (1, 2, 3) and revision (stage 6) 
concerns. 




Awareness Informational Personal Management Consequence Collaboration Refocusing 

SoC Stages 

Figure 3. Teacher B's SoC profiles in October 2009 (dotted), January 2010 (dashed), and March 2010 (solid). 
Nonintervention Group Comparison 

The changes that took place in these two teachers over the course of the 2009-2010 school year can be 
further understood in the context of the teachers who did not experience the training intervention. In 
October/November 2009, the nonintervention group (Figure 4) showed moderate interest in SMALLab (stage 0) and 
was primarily concerned with learning more about the system (stage 1) and its collaborative aspects (stage 5). By 
March/ April 2010, this group exhibited low interest in SMALLab (high stage 0), slightly increased collaboration 
concerns (stage 5), and moderate revision (stage 6) concerns. The group's remaining concerns remained largely 
unchanged. 
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Figure 4. Nonintervention group SoC profiles in October/November 2009 (dotted) and March/ April 2010 (solid). 

Discussion 




Understanding 

The SoC can be used both to understand innovation adopters at instances in time and across the lifespan of 
the change process. When evaluating single time points, one should consider the sole SoC profile along with any 
open-ended responses. Doing so will help the researcher paint a picture of and comprehend the educator's role and 
position within the adoption process. 

For instance, in October 2009, Teacher A was intensely interested in learning more about SMALLab and 
concerned about the impact that the system would have on his students. Fie also wrote of a need to incorporate more 
technology into his curriculum, as this was a part of his job function, but that he needed help doing so. At the same 
time, Teacher B was similarly interested in SMALLab and how it would impact her students. She saw herself as a 
liaison between university researchers and students. 

To understand how innovation adopters change over time, one should consider how their concerns develop, 
shift, and alternate. This can be accomplished by considering an educator's SoC profile at multiple time points. 

For example, by January 2010, Teacher A had participated in an intensive collaboration effort to implement a unique 
socio-cultural language learning experience in his classroom (Martinez et al., 2010). His concerns about the impact 
of SMALLab diminished, likely because he felt that the new activity was beneficial to students. Simultaneously, his 
desire to explore SMALLab was sustained, logistics concerns increased, and revision concerns became notable. 
These indicate that he participated extensively within SMALLab during the fall semester and carried ideas for 
improvement through to the spring. Here, Teacher B's concerns and self-described role did not change significantly 
from the previous semester, which suggests that her place in the adoption process was stable over this period. 
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Intervening 

Educators' concerns can be used to design appropriate interventions that will meet their needs. The impact 
of an intervention can be assessed by considering an educator's profile directly before and after the intervention, as 
well as in comparison to those in a nonintervention group. 

Case in point, by March 2010, Teacher A had completed the training intervention, which introduced him to 
the technical aspects of SMALLab. At this time, Teacher A's profile showcased a sustained interest in exploring 
SMALLab. His impact concerns remained low, suggesting that he was convinced that SMALLab was a positive 
influence on his students. His revision concerns were no longer in the forefront, suggesting that these concerns were 
addressed. Intriguingly, Teacher A described a significant role shift during this semester, citing the development of 
his own content, supervising the student creation of multimedia, and managing these assets in the SMALLab system. 
It is believed that his newfound focus on the technical aspects of SMALLab is a direct result of his participation in 
the training intervention. Teacher B, who also participated in the training intervention, experienced a noticeable drop 
in informational concerns, which suggests that she had learned much about SMALLab. Furthermore, her overall 
profile showed lower early stage (0, 1, 2, 3) and increased late stage (4, 5, 6) concerns. These shifts indicate 
increased experience and expertise in the use of SMALLab and are probably due to her participation in the training 
intervention. 

Simultaneously, the nonintervention group profile in March/ April 2010 was similar to the one in 
October/November 2009, with two major exceptions. First, the group's interest in SMALLab decreased sharply from 
a moderate level in October/November 2009 to a low level in March/ April 2010. Second, the group's revision 
concerns increased sharply over the same period. These factors indicate that the educators in the nonintervention 
group were becoming less interested in the innovation and that certain concerns were not being addressed. 

Meanwhile, the teachers in the intervention group maintained moderate levels of interest and decreased 
revision concerns, which suggest sustained participation in SMALLab and needs that were being addressed. These 
differences in progression through the adoption process are likely to be attributable to the intervention, as well as the 
extensive participation and collaboration efforts of the teachers in the training group. It appears that these 
experiences functioned to keep the intervention group teachers more interested and involved in SMALLab, while 
increasing their knowledge of and expertise in using the innovation. 

Conclusion 

In summary, the Stages of Concern can be an effective tool for understanding the adopters of educational 
interventions, designing interventions to meet their needs, and assessing the impact of interventions. Educators' 
concerns can be gauged at moments in time and across the lifespan of the change process using the Stages of 
Concern Questionnaire and its related diagnostic profiles. These concerns can be used to design custom-tailored 
interventions that catalyze adoption. The effectiveness of interventions can be assessed through the examination of 
individuals across multiple time points, as well as in comparison to similar individuals in a nonintervention group. In 
total, these features make the Stages of Concern a systematic way to track, understand, and support the adopters of 
educational innovations. 
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Abstract 

This study examines online technologies that can support project based learning (PBL) and how 
much use of these technologies relates to time spent on this approach to instruction, perceived 
preparedness and ability to overcome challenges. It examines the responses of 331 teachers, from 
intentionally varied types of high schools, who used PBL or similar practices to teach math, 
science, social studies or English. Findings suggest that teachers report more use of PBL, fewer 
perceived challenges, and a greater sense of preparedness when they use online technologies to 
support their practice. While use of technologies differs across school type and subjects, the 
relationship of their use to PBL use is surprisingly consistent. Results help us understand the 
prevalence of technology uses for PBL and how these are related to PBL use and perceptions, with 
implications for how new technologies might help extend the reach of PBL-related instructional 
reforms to more schools. 



Introduction 

In the last few decades there has been a concurrent growth in the availability of online technologies for 
educators and interest in supporting teacher use of constructivist-based instruction, such as project based learning 
(PBL) as a way to motivate students, engage them in "real-world" issues and learning of information age or 21 st 
century knowledge and skills. In addition, hundreds of small “start-up” high schools have been designed to 
personalize instruction and meet students’ needs. It is important to understand how these three areas of innovation — 
online technologies, PBL and small schools - overlap and to what extent teachers are taking advantage of new 
practices and technologies. 

Small schools and reform models 

In districts across the country small start-up schools have been launched and large comprehensive high 
schools have been ‘converted’ into small learning communities (e.g., Bloom, et al., 2010). The purpose of small 
school efforts has been to remove the structural barriers that impede effective teaching and learning. These schools 
are testing the idea that students learn better when they experience a climate of trust and relationships (e.g., Cotton, 
2001; National Association of Secondary School Principals, 2004; National High School Alliance, 2005). 

Evaluations of small school initiatives suggest that instruction is still a key challenge to be addressed. 

While smaller schools have been effective at creating more personalized environments for teaching and learning, 
instructional reforms have lagged behind structural and cultural changes (American Institute for Research & SRI 
International, 2005). As noted by Kahne, Sporte, de la Torre & Easton (2006), "It appears that small schools are 
fostering more personal and supportive contexts for both teachers and students but they do not appear to be spurring 
increased instructional reform” (p. 2). 

A few reform organizations have had some success pushing instructional boundaries. These organizations 
build on high-profile efforts to promote student readiness for the 21st century (President’s Committee of Advisors 
on Science and Technology, 1997), to create “break the mold schools” (Bodily, 1996). Many received financial 
support from the Bill & Melinda Gates Foundation (American Institutes for Research & SRI International, 2005) or 
were the result of legislation and funding for comprehensive school reform (CSR) intended to change “all aspects of 
schooling” (Desimone, 2002). 

The goals of many (not all) of these school reform models and the rhetoric for small schools in general, are 
"progressive" in the sense that they generally work against standardized, mechanical view of curriculum and lean 
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toward one that promotes critical engagement, interactive meaning-making, and self-realization (e.g., Feinberg, 

1999; Clinchy, 2003, Van Ryzin & Newell, 2009). A few reform model approaches have emphasized changes in the 
culture of schooling while retaining a more traditional approach to instruction (e.g., Tough, 2006; Viadreo, 2009). 

By and large, however, small schools have common origins, inspiration and goals tracing back to earlier work by the 
New American Schools Design Competition (Bodily, 1996), Coalition of Essential Schools (Meier, 1995; Sizer, 
1992) and others. 

The four model high school reform networks that participated in this study —New Tech High, High Tech 
High, Edvisions, and Envision Schools - have helped set the pace by establishing dozens of start-up schools based 
on their models. These schools have organized themselves around a consistent school design model, affiliate with a 
central organization that supports the reform model’s philosophy and practices, and form at least a minimal 
professional learning community within and across schools. 

To a large extent the reform models in our study embrace PBL as a central component of instruction 
(Pearlman, 2002; Newell, 2003). In addition, each of the models has a unique set of practices and technology 
infrastructures to help guide their work. New Tech High has a proprietary project management system and library. 
Envision Schools has a Project Exchange library and workspace for teachers, teachers in Edvisions schools have 
used Project Foundry™ software to help manage and assess student projects, while High Tech High has established 
a “digital commons” for sharing resources among teachers. 

Project based learning 

Project Based Learning (PBL) is a constructivist-based instructional approach that is designed to support 
more engaged learning (Duffy & Jonassen, 1992; Brooks & Brooks, 1993). This approach uses “projects" as 
vehicles to encourage student motivation and to provide a means for demonstrating and explaining what they have 
learned. This approach has much in common with problem-based or inquiry-based instruction (Barron & Darling- 
Hammond, 2008; Savery, 2006). Evaluations of small school initiatives have highlighted the prevalence of this 
general approach: 

“Among the schools in this initiative that reported efforts to implement a common pedagogy 

across all classes, project-based learning (PBL) is the most commonly cited instructional strategy . 

. . .in practice, many educators will refer to the same activity interchangeably as ‘project-based’ or 

‘problem-based’ learning, or simply ‘PBL’.” (Mitchell, et al., 2005, p. 40). 

Evidence of growth of interest is provided by the growing number of teachers who have received PBL 
materials (our organizational database is almost 10 times larger today than when we undertook this study), the 
growth of web sites that emphasize PBL as a core instructional concept (e.g., Edutopia, 2001), inclusion of project- 
based learning in policy documents such as from the National Middle School Association (Yetkiner, Anderoglu, 
Capraro (2008) and the National High School Center (Harris, Cohen & Flaherty, 2008, p. 3), and even state-wide 
efforts, notably in West Virginia (Williamson, 2008) and Indiana (Indiana University, 2010). 

No two teachers implement PBL in the exact same way, so this makes it difficult to define exactly what 
PBL is and then study its effectiveness. Over the years, research on PBL has expanded from a specific kind of 
problem-based learning in medical schools to a wider variety practices, subjects and grade levels (Walker & Leary, 
2008). Taken as a whole, with the exception of memorization for short-term learning, PBL use has been shown to 
be as effective as traditional instructional approaches, and in many studies superior (Buck Institute for Education, 
2010; Edutopia, 2001; Geier, et. ah, 2008; Strobel & van Barneveld, 2009; Walker & Leary, 2009). 

Challenges to PBL 

Implementing PBL and other inquiry-based instruction can be challenging, requiring changes in classroom 
management and forcing teachers to be ready with a vast array of resources and knowledge (Blumenfeld, Soloway, 
Marx, Krajcik, Guzdial, & Palincsar, 1991; Mergendoller, Markham, Ravitz & Larmer, 2006). Teachers can report 
difficulty or feel under-prepared when making the transition to this more student-centered approach (Ertmer & 
Simons, 2006; Marx, Blumenfeld, Krajcik, & Soloway, 1997). On the other hand, when PBL is well designed and 
well supported, it is possible for teachers to be surprised by what their students can accomplish (e.g., Johnson, 

Smith, Smythe & Varon, 2008). 

It does not seem reasonable to expect teachers to learn about and use this approach entirely on their own, 



178 



although some certainly do. Effective use of PBL requires extensive planning and professional development for 
teachers, a supportive environment, and tools and strategies for both teachers and students (Hmelo-Silver, Ravit & 
Chinn, 2007). If instructional reforms have had limited success in small high schools, this is likely due to challenges 
associated with how much preparation is needed to design and implement PBL effectively. 

The Role of Technology - Untangling a Tangled Relationship 

It is important to understand how much online technologies might be helping provide valuable PBL 
resources for teachers who are exploring PBL use. Teachers who want to use PBL and must confront the above 
challenges can start by gathering additional ideas and resources by going online. They can locate models or 
examples in their subject area or find tools that can be useful for coming up with ideas and planning projects. 
Examples of online features considered in this study include tools for linking to outside experts, mentors, or other 
schools, collections of projects, tools for managing or assessing student work, for planning and designing projects, 
and more general features such as blogs, WIKI, and others. 

If teachers are already familiar with PBL, online technologies can help scaffold instruction and provide the 
kinds of supports that teachers and students need to be successful with PBL (e.g., tools for collaboration, feedback, 
managing work, etc.). Research suggests that both of these elements — teachers’ development of PBL-related 
knowledge and the availability of implementation scaffolds — are critical to the implementation and effective use of 
PBL (Boss & Krauss, 2007; Cognition and Technology Group at Vanderbilt, 1992; Ertmer & Simons, 2006; Hmelo- 
Silver, 2006). 

As a result of the Internet explosion many simply assume new technologies will play an important role 
supporting PBL (more than in the past, when connections were slow, resources were not widely available, etc.). As 
one reviewer of an earlier version of this paper noted “It seems implausible to suggest that PBL should be 
implemented without the use of online resources.” 

In some ways, it does seem implausible to not use new technologies. A convergence of interest in PBL use 
and new technologies has led to many interventions that intentionally incorporate technology as a key component of 
PBL use (e.g., Saye & Thomas, 2005). Many technology-related barriers have come down and opportunities for 
teachers to conduct projects in a technology -rich environment have never been greater. However, there is still a lot 
to be learned about how and in what ways teachers incorporate technology and PBL use in their practice. 
Importantly, because our definition of PBL does not require online technologies, we can see how PBL use varies 
with and without their use. 



Research Questions 

This study explores the relationship between PBL-related teaching practices and use of online features to support 
these practices. It addresses the following questions: 

For teachers across different high school contexts and academic subjects. . . 

What is the prevalence of PBL' use, preparedness and challenges? 

What is the prevalence of online feature use to support PBL? 

To what extent is online feature use related to preparedness, challenges, and use of PBL? 

As noted for Becker & Lovitts, (2003), the only way we can hope to see how new technologies contribute 
to teaching and learning is to explore their use independent of pedagogy. Once we accept that some PBL-using 
teachers and projects do not require online technologies (e.g., for projects that involve designing a golf course for 
the moon, staging a debate about the death penalty, observing changes in bird populations, presenting a land use 
policy), it becomes possible to explore how use of technologies is related to differences in PBL use and to examine 
the apparent convergence of PBL and use of new technologies more closely. 

Methods 



Population and Sample 

In the fall of 2007, over 400 PBL-using teachers completed an online survey -- a 35% response rate based 
on a sample of approximately 1200 teachers. This included teachers in the current analyses - 33 1 teachers who 
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taught academic subjects (math, science, social studies or English) in public high schools and confirmed that they 
used PBL in these subjects. These teachers came from diverse school types that we categorized as large 
comprehensive high schools (n=61), small schools and small learning communities (n=104), or one of the four 
reform networks New Tech High, High Tech High, Edvisions, and Envision Schools (n=166). To avoid biasing the 
results, we analyzed data separately for each of these school types. 

The sample is considered to be fairly representative ofPBL-using teachers, clearly representing a variety of 
schools, academic areas and geographic locations. Mid-west and rural states were under-represented as were a few 
key states in the south. In addition, a couple east coast organizations were not included whose participation could 
have boosted the number of PBL using teachers substantially, notably Big Picture Schools (Littky & Grabelle, 2004; 
McDonald, Klein, Riordan & Broun, 2003) and Expeditionary Learning Outward Bound (1999). 

The sample of probable PBL users was based on information about sales of PBL materials and attendance 
at workshops. In addition to querying our own organizational database and searching for teacher lists on the web, 
this study received support from reform model staff who provided lists of teachers or schools receiving PBL 
professional development or materials, and professional development staff from the Center for Effective School 
Practices (with workshops held in NJ and OH), the San Diego Renewal High Schools initiative, and the North 
Carolina New Schools Project (see Ravitz, 2008 for full sample details). Each teacher received multiple contacts 
and both a social and economic incentive, with communications closely following strategies outlined by Dillman 
( 2000 ). 

Measures 

The survey instrument was constructed after reviewing existing surveys and piloting different version of 
questions in 2006. Development of the instrument used methods similar to “cognitive interviews” (Desimone & 
LeFloch, 2004) and revising questions until they appeared to generate reasonable responses from teachers in vastly 
different kinds of schools. 

Project based learning {PBL) was defined for participants minimally as an approach to instruction that a) 
features in-depth inquiry b) occurs over an extended period of time like a week or more, c) is student/self-directed to 
some extent, and d) requires a formal presentation of results. While there are other characteristics of PBL we might 
have included, these criteria represent a minimal definition that allows variation in other aspects like group work or 
technology use. Participants were invited to substitute a preferred term for PBL, as long as it included the above 
characteristics. Approximately 1 7% said they preferred problem-based, inquiry-based, or some other term. 

In addition to be provided with the above definition, teachers saw a list of example types of projects (e.g., 
researching a community issue and creating an action plan, making observations and collecting data, creating a 
museum-like exhibit, etc.) so they would know what kinds of practices were being referenced. They were then 
instructed to pick the academic course in which they used the most PBL and report on their teaching of that course. 

Time spent using PBL was based on teachers’ response to the following item: “For a typical student in this 
course, how much of their overall TIME was spent on project based learning?” scored on a 6-point scale (1 = none 
or almost none, 2 = less than Vi, 3 = about Vi, 4 = about 14, 5 = about %, 6 = all or almost all). 

Online feature use was based on a count of the number of features that were used at least “a little”, because 
not many responded that they used any feature “a lot”. The online features we asked about are listed below, with the 
accompanying prompt “For each of the following Internet-based features of capabilities, indicate whether you have 
seen or used this kind of online resource or tool for conducting PBL.” 

• Online collaboration tools (e.g., blog, Wiki, listserv, social networking) 

• An online collection of high quality projects 

• An online collection of PBL resources (e.g., rubrics, templates, examples, descriptions, suggestions, video) 

• Tools created to help you or your students design and manage projects online 

• A way for your STUDENTS to post work to get feedback or be assessed by you or others 

• A way for YOU to get feedback from other teachers or adults on your projects or student work 

• Tools for linking you or your students to outside experts, mentors, or other schools 

Instead of focusing on specific web sites or software platforms, we described more general features of 
online use (e.g., for planning, giving feedback, collaborating, etc.) as might be provided by a variety of web sites or 
tools. We limited the features we examined to ones that teachers in our pilot study said they used in planning PBL 
or that helped them with PBL implementation. These online features (available using multiple software or web site 
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configurations) provide the capacity to conduct projects or support them. They allow teachers to learn about PBL 
and manage projects more effectively, to share PBL examples, experiences and advice. They also may make it 
easier for communication and feedback to occur between teachers and students, or across different schools. In each 
of these ways, the features can be expected to play a role in supporting PBL use. These online uses are probably 
related to use of more general online features (like search engines, email, or file sharing) and other computer uses 
that were not asked about because we wanted to focus on specifically PBL-related tools and practices. 

Perceived challenges for PBL use was assessed using five items that were determined through 
conversations with teachers to be critical for the effective use of PBL (e.g., I lacked models or examples for using 
PBL in my subject area with my students”). Items were scored on a 4-point scale (ranging from 1 = not a challenge 
to 4 = a major challenge). 

Preparedness for PBL was assessed using nine items that asked teachers how prepared they felt to carry out 
tasks related to effective PBL use (e.g., “To what extent do you feel prepared to assess individual student’s content 
learning using PBL”). Items were scored on a 4-point scale (ranging from 1 = not at all prepared to 4 = very well 
prepared). An index based on the mean of all nine preparedness items had strong reliability (standardized alpha = 
.91), while the index for challenges was reliable enough for our purposes (standardized alpha = .80). 

Finally, the kind of school was considered in our analyses and controlled for accordingly. Small schools 
and learning communities were defined as having less than 500 students and identified as being part of a reform 
model network or not. 

Analysis Plan 

Descriptive data helps us understand the prevalence of responses across a wide range of schools. Differences in 
mean scores used standardized z-scores, and ANOVA statistical tests, while percent differences used cross- 
tabulations and chi-square tests. The research questions use correlations between online feature use, perceived 
challenges, perceived preparedness and time spent on PBL with data presented using cross-tabulations and mean 
comparisons controlling for school type or subject taught. 

Results 

Although school context and subject limit use of PBL and online features, online feature use is clearly most 
prevalent among the most frequent PBL users. We see this in all three types of schools we studied. The more 
teachers used online features the more prepared they felt and the better they were able to handle PBL-related 
challenges. Some key descriptive findings provide an indication of differences across school types. The descriptive 
findings are not particular surprising, but they provide interesting contrasts regarding teachers in small schools that 
were and were not affiliated with a reform model. Overall, the most PBL use and the greatest amount of technology 
use were seen in the reform model schools, where teachers reported fewer challenges and greater PBL preparedness. 

o 83% of reform network teachers reported spending more than “about !4” of their time conducting PBL, 
compared to 48% of teachers in other small schools and 34% in the larger schools. 

o The percentage of reform network teachers who reported feeling “well prepared” for PBL was higher than in 
other schools — with statistically significant differences in preparedness for meeting state or district standards, 
assessing content learning and planning or designing new projects. 

o Professional development for PBL was rarely a challenge in reform model schools and other small schools 
(reported by 17% in both), but was frequently a challenge in larger, comprehensive schools (reported by 40%). 

Some key differences in features used (as shown in Figure 1) include the following: 

o More than half (56%) of the teachers in reform model schools had students post their project work and receive 
feedback, compared to 21% and 18% of teachers in the other two types of schools 

o More than half (58%) of the teachers in reform model schools had designed and managed projects online 
compared to 36% and 31% of teachers in the other school types. 

o Online collections of PBL resources were prevalent in reform model and unaffiliated small schools (reported by 
about 60% of each group), but online tools to plan and manage projects were only prevalent in the reform model 
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schools (58% vs. 3 1% of other small schools) 

Figure 1. Use of online technology features for PBL, by school type 



Onlne collaboration tools, e.g.. 
blog. Wiki. Iistserv. social 
networking, etc). 

Tools for Inking you or your 
students to outside experts, 
mentors, or other schools 

Away for your STUDENTS to 
post work to get feedback or be 
assessed by you or others 

A way for YOU to get feedback 
from other teachers or adults on 
your projects or student work 

Tools created to help you or your 
students design and manage 
projects online 

Online collection of PBL 
resourcos (eg., rubrics, 
templates, video) 

Oflflne collection of high quakty 
projects 




o» 



UP* 



73 * 



These descriptive findings reinforce the idea that the most extensive use of PBL and supporting 
technologies are in the reform model schools, followed by the other small schools. This would indicate that overall, 
across school types, technology use is associated with PBL use. In fact, the overall correlation is 0.37 (p < .05). 

The next section examines a more significant question, whether this pattern exists even within school types. 

Relating PBL use to online feature use 

Findings clearly indicate that the more teachers used online features the more they used PBL, the more prepared 
they felt and the better they were able to handle PBL-related challenges (Table 1). The largest correlation with 
preparedness was in small schools outside the reform model networks (r=,46) but the correlation was positive for all 
three types of schools. The correlation between PBL use and online features was .33 for teachers in reform model 
schools, and .23 for teachers in other small schools (both p < .05). For larger schools, the correlation to time on 
PBL was not ssignificant, but feature use was significantly correlated to both preparedness and fewer challenges. 

Table 1. Correlation of features used to time on PBL, preparedness and challenges, by school type. 

Correlation of number of features to. . . 



Correlations for. . . Time on PBL Preparedness Challenge 



All schools combined (n=330) 


.37* 


.35* 


-.18* 


Reform Model (n=161) 


.33* 


.25* 


-.06 


Small Schools (n=103) 


.23* 


.46* 


-.13 


Large comprehensives (n=61) 


.10 


.28* 


-.25* 



*p < .05 

Table 2, below, presents significant differences in the percentage of teachers who reporting that they felt “well 
prepared” to handle common PBL-related instructional tasks by use of online features, analyzed separately for each 
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school type. The comparison is between teachers who reported that they used the online feature “at least a little” 
versus those who reported that they did not use that feature within a particular school type. 
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Find existing projects S (80%, 51%) S (82%, 50%) S (79%, 51%) S(82%,45%) R (71%, 56%) S (69%, 49%) R(76%, 58%) 

that are high quality S (70%, 46%) 

L (68%, 34%) 

R=Reform network (N=164); S =Small school / small learning community (N=105); L=Large, comprehensive school (N=61) 

All comparisons are statistically significant, chi-square p < .05 Features are slightly abbreviated from the actual items as shown in the methods section above. 




Preparedness and Feature Use 



All significant differences in were in the expected direction with feature use relating to a higher 
percentage of teachers who felt “well prepared” and a lower percentage reporting major/moderate 
challenges (Table 2). 

• In larger, comprehensive schools, use of a feature “to get feedback from other teachers or adults” was 
related to statistically significant differences five of the nine PBL preparedness tasks 

• Use of tools created “to help you or your students design and manage projects online” was related to 
teachers’ sense of preparedness in unaffiliated small schools across all nine tasks. 

o For example 82% who used this tool felt well prepared to “promote depth or quality” in PBL, 
compared to 53% of those who did not use this kind of tool. 

• Use of tools for “linking you or your students to outside experts” related to significant differences in 
preparedness across all nine tasks for at least one of the type of school. 

o For example, 90% of teachers in unaffiliated small schools who used this feature felt well 
prepared to assess student content learning in PBL, compared to 50% who did not use that 
feature 

• In large schools, use of a tool for “feedback from other teachers or adults on your projects/student 
work” was related to preparedness for five of the nine preparedness tasks. 

o For example, 71% of the larger school teachers who used “an online collection of high quality 
projects” felt “well prepared” to plan and design new projects, compared to 46% of teachers 
in those schools who had not used this feature. 

Challenges and Feature Use 

We see a similar pattern, although less extensive, regarding challenges for PBL. The challenge 
that was most substantially reduced with use of online features concerned “lack of professional 
development or coaching in PBL”. Some key differences within school type include the following: 

o In reform model schools, 8% who used online tools to manage student projects cited lack of 
professional development as a challenge, compared to 30% who did not. 

o In reform model schools, 8% of those who had a way to get online feedback said professional 
development was a challenge, compared to 24% who did not use this feature. 

o Only 7% teachers who used tools for linking to outside experts felt lack of professional development 
posed a major or moderate challenge, compared to 22% of the teachers who did not use these tools. 

o Less than one-fifth (18%) of teachers in large comprehensive schools who used online tools to design 
or manage projects said professional development was a challenge, compared to more than half (53%) 
that did not use such tools. 

The online feature that related to differences in perceived challenge across all five issues was 
“tools created to help you or your students design and manage projects online”, although the type of school 
for which the significant difference occurred varied. 
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Across Subjects Taught 

A final set of analyses concerns the correlation of the number of online features to the measure of PBL 
use measure and the challenges and preparedness indices, by subject taught. For these analyses we used 
correlations or compared mean z-scores on indices. Statistically significant (p < .05) differences by subject 
taught included: 

• Among science teachers online feature use is associated with more PBL use (r=.36), a decrease in 
mean challenges (r=-,34) and increase in mean preparedness (r=.51). 

• Among social studies teachers there was a strong relationship between the amount of PBL use and the 
number of online technologies used among social studies teachers (r=,57, p < .001) 

• English teachers who used more online features felt better prepared for PBL use (r=.33, p < .01). 

• Math teachers reported the least PBL use overall (mean z score = -.35) and felt least prepared for PBL 
use (mean z score= -.31), but there was a strong correlation between online feature use and PBL use 
(r=.60, p < .001). 

Although these subject-specific relationships may be largely spurious - due to the fact that we did not 
have enough cases to control for school differences at the same time - it is useful to see similar patterns 
data across subject, indicating that the relationship is widespread, not limited to certain disciplines. 

Discussion 

This study is important because it concerns efforts to improve instruction by using online 
technologies and PBL. Regardless of school type, effective design and use of PBL requires ongoing 
professional development and support. Even in the best of circumstances, even in schools that have fully 
embraced PBL, the professional development and resources needed for effective use of PBL are unlikely to 
be fully met by most schools (Ravitz, 2010). Results suggest that online tools may provide an important 
way for teachers and schools to help address the challenges of PBL use. 

It is helpful to know which technologies and practices have proliferated and which are only seen in 
reform model schools. For example, it appears far more teachers have used online libraries of projects and 
resources than have used tools to help with designing and managing projects. The latter feature is seen 
more frequently in reform model schools and is closely associated with PBL use and preparedness. 

Sometimes teachers in non-network small schools (i.e., unaffiliated with reform models) reported 
similar practices or perceptions as reform model teachers, but not always. For example, in both types of 
small schools there appeared to have been ample professional development for PBL (with this being a 
challenge for only 17% in both school types). However, teachers in unaffiliated small schools reported 
that time in the curriculum for PBL use was much more frequently a challenge. This suggests that PBL is 
emphasized to a large extent even in unaffiliated small schools, but conditions and practices in the reform 
models are different. 

Across school types, the technology feature that was most closely associated with PBL 
preparedness varied. This suggests certain technology applications are useful in different kinds of schools. 
For example, in larger comprehensive high schools the feature that best predicted preparedness was a way 
for teachers to get feedback from other adults, perhaps indicating a lack of effective feedback mechanisms 
in larger schools. In the reform model schools, the most predictive feature was access to outside experts to 
support projects, perhaps reflecting efforts to make PBL more authentic for students. In unaffiliated small 
schools access to outside expertise was also a predictor of preparedness, but so was using online tools for 
designing and managing projects. Perhaps without a reform model to help guide them, teachers in small 
schools may have to rely more on tools they find on the Internet to help them design and manage PBL. 

To conclude, there is a robust relationship between use of online technologies and greater amounts 
of PBL use and preparedness. The correlation between online features and PBL preparedness was 
significant for numerous study strata, not just the three kinds of schools. However, the study data cannot 
answer questions about the causes of this relationship. The direction of causality that seems most important 
is how online technology use increases or improves PBL use. However, there is almost certainly a 
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mutually reinforcing relationship wherein technology helps teacher implement PBL but PBL also helps 
teachers integrate technology by providing reasons for its use. 

It would be interesting to see more studies looking at how technology use differs when PBL is 
used, as well as how PBL differs when technologies are used, including perhaps technologies that we did 
not include -general online search features, or productivity software. However, the goal of promoting 
technology use seems less important than the goal of promoting effective PBL use. Educators are not in the 
business of coming up with uses for technology. As Ely famously asked “Technology is the answer, but 
what was the question?” In this case, how to improve preparedness for PBL use seems to be a question 
worth asking, and online technologies seem to provide part of the answer. 

Although the experiences of individual teachers and schools may differ, the overall pattern suggests 
that technologies are probably helping teachers increase their PBL use and preparedness. Future research 
might focus on how reform model teachers are using online tools to support their PBL use and on finding 
ways to share these lessons and tools with other small schools to see if more teachers can be prepared to 
use PBL effectively. Clearly technology supports are only part of the answer to effective PBL use, but 
increased used of online technologies may be one way that instructional reforms like PBL can be 
disseminated from reform model schools to others. 
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Abstract 

Over the years, supporters of project- or problem-based learning (PBL) in K-12 have generally had to rely 
on limited research, weak research methods, and sometimes mixed results. Results from a federally 
funded, randomized controlled experiment in high school economics provide evidence for the efficacy of 
PBL and should fuel growing interest in this approach to instruction. The study examined the impact of a 
one-week summer professional development institute and use of five PBL curriculum modules on the 
economic knowledge of approximately 7,000 12th grade students, taught by 76 teachers in 66 schools. 
Student outcomes that were studied included scores on the standardized Test of Economic Literacy (TEL), 
and scores on performance assessments of student conceptual understanding. Teacher outcomes included 
confidence in teaching economics and satisfaction with teaching materials. The findings, prepared by the 
Regional Educational Laboratory West (REL West) at WestEd, indicates there were significant positive 
impact for students of teachers who received the curriculum compared to their peers and teachers scored 
higher in satisfaction with teaching materials and methods than those in the control group. 

Purpose 

This study was undertaken to provide strong experimental evidence concerning the effectiveness 
of PBL use in high school economics. The study was designed to assess student-level impacts of a 
problem-based instructional approach using a randomized controlled trial. It tested the effectiveness of a 
series of five curriculum units developed by the Buck Institute for Education (www.bie.org) on student 
learning of economics content and problem-solving skills. This study targeted high schools in urban, 
suburban and rural areas and engaged teachers who committed to teach economics for two semesters during 
the 2007/08 academic year. 

Student achievement outcomes were the study’s primary focus. Specifically, it examined whether 
the Buck Institute’s curriculum improved grade 12 students’ content knowledge as measured by the Test of 
Economic Literacy (TEL), a widely accepted, standards-aligned test used across the United States, along 
with students’ problem-solving skills in economics as measured by a performance -task assessment. In 
addition to these student outcomes, the study examined changes in teachers’ content knowledge in 
economics, their pedagogical practices, and satisfaction with the curriculum. 

Why study economics instruction? 

For decades, economists, prominent educators, and business and government leaders have 
advocated for developing economic literacy as an essential component in school curricula. Their arguments 
have ranged from the need for improving the ability to manage personal finances to the value of economic 
education for critical thinking and an informed citizenry. “The case for economic literacy is obvious. High 
school graduates will be making economic choices all their lives, as breadwinners and consumers, and as 
citizens and voters. A wide range of people will be bombarded with economic information and 
misinformation for their entire lives.” (Tobin, 1986). 

At the federal and state levels, economics has received increasing attention as a critical content 
area for K-12 education. Forty-eight states now include content standards in economics, with 40 requiring 
their implementation, and 17 requiring a course in the subject for graduation. (National Council on 
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Economic Education 2007). Economics is a required course for high school graduation in California and 
Arizona, the two states where the study takes place, and is usually taught to seniors for one semester. 

While there is growing agreement on the need for some economics content in K-12 education, there 
is less consensus about where it fits into the curriculum, how to teach it effectively, and how much subject 
area background should be required of classroom instructors. Additionally, there is concern that many 
students do not understand economics and their teachers may lack content knowledge, access to relevant 
teaching materials and adequate professional development (Walstad, 2001, Watts, 2006). 

Why study PBL use? 

PBL is an approach to instructional design that can help organize the curriculum and deliver 
instruction. It provides a mechanism to gain student attention, to motivate and anchor learning. It provides 
opportunities for authentic assessment of content and skills, and use of new technologies. Research 
suggests that for the most important outcomes (other than short-term recall or recognition), such as long 
term retention and assessments involving elaboration of understandings, PBL has been as effective as 
traditional instructional approaches, and there are many studies that show PBL to be superior (Strobel & 
van Barneveld, 2008; Walker & Leary, 2008). Of these studies, relatively few have been well-designed K- 
12 experiments, however a few studies do suggest PBL can be effective in diverse K-12 settings (Barron & 
Darling-Hammond, 2008; Boaler; 1992; Edutopia, 2001; Marx, et ah, 2004). 

There has been an evolution of thinking about what PBL is. PBL can no longer be equated with 
“minimally-guided or “discovery learning’’ (Hmelo-Silver, Duncan & Chinn, 2007: Mayer, 2004; 
Kirschner, Sweller & Clark, 2006). Gradual consensus appears to be building that not all PBL is created 
equal. Considerable effort has to go into problem or project design, scaffolding and management of 
learning activities, and the requisite professional development for teachers in order for PBL to be effective. 
As Dewey noted, “The belief that all genuine education comes about through experience does not mean 
that all experiences are genuinely or equally educative.” (1938, p. 25). Today’s projects, such as those 
conceived at High Tech High School (http://www.hightechhigh.org/projects/), Envision Schools 
(http://www.envisionprojects.org), Expeditionary Learning schools (http://elschools.org/), or New Tech 
Network schools (http://www.newtechnetwork.org/) look nothing like the “child-centered” progressive 
education ideas of William H. Kilpatrick (1918), who popularized the ‘Project Method” in the early part of 
the 20 th century. 

Contemporary PBL is standards-focused, and can incorporate a wide variety of instructional 
strategies -even more “traditional” lectures, skill-building activities, library and Internet research, and 
various forms of writing. Students can be assessed by both performance -based and traditional means. PBL 
differs from a traditional curriculum approach, however, by creating a reason to learn or “need to know” 
(besides getting a good grade on the test) prior to using any of these other techniques. PBL also provides 
opportunities for students to develop (and be assessed on) their ability to work together, manage complex 
work, explain themselves to an external audience, etc. Well-conceived projects or problems provide 
questions and challenges that can serve to organize the curriculum as a whole. These characteristics of 
PBL are reflected in the definition of PBL as “a systematic teaching method that engages students in 
learning essential knowledge and life-enhancing skills through an extended, student-influenced inquiry 
process that is structured around complex, authentic questions and carefully designed products and tasks” 
(Mergendoller, et ah, p. 587). 

Evaluations of small school initiatives have highlighted the prevalence of PBL: 

“Among the schools in this initiative that reported efforts to implement a common 
pedagogy across all classes, project-based learning (PBL) is the most commonly cited 
instructional strategy ... .in practice, many educators will refer to the same activity 
interchangeably as ‘project-based’ or ‘problem-based’ learning, or simply ‘PBL’.” 

(Mitchell, et ah, 2005, p. 40). 

Additional evidence of growth of interest in PBL includes the number of schools and districts 
using PBL as a key component of school wide reform (Ravitz, 2010), state-wide initiatives (Williamson, 
2008; Indiana University School of Education, 2010), web sites that emphasize PBL (e.g., Edutopia, 2001), 
and policy documents such as from the National Middle School Association (Yetkiner, Anderoglu, Capraro 
(2008) and the National High School Center (Harris, Cohen & Flaherty, 2008, p. 3). 
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A PBL approach to economics 

The approach to PBL in this study represents a modification of the problem-based approach 
originally developed for use in medical schools (Mergendoller, Maxwell, and Bellisimo 2000). The 
modifications were made for use in US high school settings, although the resulting curriculum has been 
found to be effective in college courses (Smith & Ravitz, 2008) and when translated for use overseas (Mo 
& Choi, 2003). In contrast with the more typical textbook and lecture- driven approach that is commonly 
seen, teachers initiate inquiry based on examination of real-world economic problems that require a set of 
disciplined, analytic steps. The intent is that students learn to contextualize, understand, reason, and solve 
problems using analytic skills that are taught as part of the curriculum. 

Each of the five curriculum modules used in this study is set up around a fictitious (simulated) 
case study that is well suited to student-driven problem solving. Each unit is organized around four major 
phases: 

1. Project launch 

2. Framing the inquiry 

3. Problem-solving and learning activities 

4. Presentation, assessment and debrief 

The step-by-step teaching guide that is provided is the cornerstone of each module. It lays out for 
teachers the problem statement, introduction, placement in curriculum, concepts taught, objectives, content 
standards, time required, lesson description, resource materials, sequence of the unit, procedures, and do’s 
and don’ts. For the PBL group teachers, the curriculum is introduced during a five-day professional 
development workshop led by expert teachers who have used the materials extensively in classrooms. 

The logic for the intervention and its intended effects are outlined in the below. Student 
achievement outcomes are of primary importance. These are hypothesized to be mediated by changes in 
teacher knowledge and pedagogical practice. 



Teacher Professional 
Development 



Content 

Review of economic 
theory and related 
applications; 
curriculum review 
Pedagogy 
Problem-based 
approach to 
teaching and 
support strategies 
for students 



Classroom 

Instruction 



Curriculum 

Fluent presentation 
of curriculum; 
scaffolding for 
students 
Engagement 
Problem-solving 
strategies that build 
economic content and 
reinforce core 
analytic thinking 



Student 

Performance 



Concepts 

Broad-based literacy 
in micro- and 
macroeconomic 
concepts 

Problem Solving 
Ability to 
apply concepts 
and analytic 
approaches to 
real-world problems 



The intervention for this study included a combination of materials and professional development. 
The professional development consisted of a 40-hour economics course for teachers, held over five days in 
the summer of 2007. This workshop introduced five curriculum modules that contained fundamental 
components of the curriculum standards in economics in the states where the study was implemented 
(Arizona and California) and covered approximately 50-70 percent of the curriculum content of economics 
classrooms. Teachers left the session with an understanding of how to sequence the material for each 
curriculum unit into consistently applied teaching steps. During the course of the semester, periodic 
videoconferences were offered to support teachers during the implementation of the units, although these 
were sparsely attended. 
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Research Questions 



The research questions for the study reflect the expected changes in teacher knowledge and practice, 
attitudes toward teaching economics and student outcomes. They were as follows: 

1. Does PBE change students’ content knowledge in economics? 

2. Does PBE change students’ problem-solving skills in economics? 

3. Does PBE change teachers’ content knowledge of economics? 

4. Does use of PBE change economics teachers’ instructional practices? 

5. Does the use of PBE change teachers’ satisfaction with teaching materials and methods use? 

Methods 



The study population and sample 

Implemented from summer 2007 to spring 2008, the study targeted high schools in rural, 
suburban, and urban areas of both California and Arizona, two states where economics is a required course 
for graduation. Study participants included economics teachers who were randomly assigned to an 
intervention or control group. 

Every school in Arizona and California with enrollment of more than 1,500 students 
(approximately 1,000 schools) was contacted to discuss the study. The recruiter had some discussion with 
administrative staff or teachers in nearly all of them. The resulting pool of schools in the study sample was 
106. Only after a teacher was found willing and eligible to participate in the study were the school and 
district asked to permit study participation. Thus, the recruited sample was composed of teachers who 
volunteered to participate in a randomized controlled trial and who committed to participate in the 
professional development and to implement the curriculum if randomly assigned to the intervention group. 
The study team was not able to collect information about teachers who declined to participate in the study, 
and as a result, it is unable to make any inference about the differences between teachers who did and did 
not agree to participate. 

After random assignment, and accounting for attrition and missing data, the intervention group for 
Spring 2008 included 35 teachers and their 2,502 students, while the control group included 29 teachers 
and their 1,848 students. There was attrition of 12 control teachers and 7 intervention teachers who did not 
return data for this study. In addition, two teachers in the treatment group did not provide outcome 
measures on the performance tasks. The number of missing students for each measure ranged from 1 1% to 
15%, except for the performance tasks which were missing for 19% of the control group and 23% for the 
treatment group students. Students were excluded from the impact analyses if they were missing associated 
outcome measures. 

Teachers in control schools participated in their regular annual professional development activities 
and continued their usual instructional practices in economics classrooms. (As an incentive for their 
participation they received the curriculum and professional development in the year following the study). 

Measures 

The experimental study was designed to test whether problem-based instruction in high school 
economics can result in gains in students’ content knowledge. The primary outcome measure for this study 
is content knowledge gains for students in economics measured by the Council for Economic Education’s 
Test of Economic Literacy (TEL), a 40-item closed-response economics exam (Walstad and Rebeck, 

2001). This is a widely accepted, standards-aligned test used across the United States to measure economic 
literacy among high school students. 

The research team augmented this outcome measure with an opportunity to test students’ abilities 
to reason and solve problems with the concepts they had learned. Student problem-solving skills are 
measured with open-response performance assessments of applied economics concepts (performance task 
assessments) developed by the Center for Research on Evaluation, Standards, and Student Testing at the 
University of California, Los Angeles (UCLA CRESST). These written tasks gave students the ability to 
demonstrate problem-solving skills as they answered open-ended essay question, as outlined by Baker, 
Aschbacher, Niemi & Sato (1992) and Niemi (1996). 
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The assessment tasks were created and piloted by CRESST with more than 300 students in spring 
2005, prior to this study. These feature paper-and-pen thinking and writing responses based on contextual 
prompts that focused on monetary policy/federal funds, monetary policy/employment, fiscal policy, 
consumer demand, and opportunity costs. These tasks were chosen because of their focus on fundamental 
economics concepts and their alignment with state standards in the course. These economics performance 
assessments do not explicitly reference the Buck Institute’s curriculum and were piloted both with teachers 
who used the relevant curriculum units and with teachers who did not. The assessment tasks and their 
common rubric were revised based on several rounds of student responses. Based on this initial work, 
CRESST indicated that the tasks provide good evidence of the quality of student conceptual understanding 
in economics. The performance tasks were administered at the end of each semester as a measure of student 
learning. (These assessments did not have a pretest component.) Each task required 15-20 minutes to 
complete (75-100 minutes for all five tasks). 

Pedagogical practices were measured using teacher self-reports on 9 items using items similar to 
those used by Ravitz, Becker & Wong (2000) to indicate the frequency different practices were used. 
“During the past semester, how often did you give assignments in economics that required students to do 
the following?” [1 = Never, 2 = A few times, 3 = Once or twice a month, 4 = Once or twice a week, 5 = 
Almost every day]: Work on projects that take a week or more; Work together in small groups; Use a 
rubric to help assess and guide their work; Organize and analyze information or data; Come up with 
solutions to economic problems, like those found in the real world; Consider alternative solutions to an 
economic problem; Orally present their work or ideas to others; Use the Internet to get information; and 
Use computers — besides word processing — to analyze or present data (such as Excel). The index measure 
based on these items was reliable, coefficient alpha=,85. 

Teacher satisfaction with the curriculum was addressed by asking how satisfied on a scale from 1- 
5 (from “very unsatisfied” to “very satisfied”) teachers were with the curriculum materials you have for 
teaching economics, and the methods you use to teach economics, with the mean on the two measures 
being reliable, coefficient alpha=,80. 

Data Analysis 

The analysis for this study compared the outcomes for students and teachers in the intervention 
group with their counterparts in the control group after the students’ economics course had been completed 
in spring 2008. All outcome variables were treated as continuous variables in the impact analyses 
(estimated using multilevel or single-level linear regression models). To increase the precision of the 
estimates, a set of baseline characteristics of students and teachers was included in the models as 
covariates. These included student gender, race/ethnicity and pretest measures, as well as teacher 
background indicators (years teaching, number of college courses in economics and confidence teaching 
economic concepts) and aggregated student pretest measures. Additional information about dummy 
variables and randomization procedures are available in the full report, along with HLM models and 
treatment of fixed and random effects (Finkelstein, et ah, 2010). Statistical power estimates for Type 1 
error = .05 indicate a minimum detectable effect size of .18-.21 at the student level and .55 at the teacher 
level (based on 83 teachers and 40 students per teacher) 

Findings 

This REL West study was designed as an in-school, randomized controlled trial that tested the 
effectiveness of a Problem Based Economics (PBE) curriculum developed by the Buck Institute for 
Education on student learning and problem-solving skills. The study found that students benefited from the 
combination of the curriculum, the associated professional development program, and the support that was 
provided as part of the study’s implementation. The students whose teachers used the problem based 
curriculum in their classrooms scored significantly higher on both content assessments (TEL and 
performance tasks) compared to students who were not exposed to the curriculum in their economics 
classes. In addition, teachers who used the curriculum were more satisfied with the materials than those 
who used standard teaching materials. 
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Student benefits 



Results indicate that students whose teachers had received professional development and support 
for use of the PBL economics curriculum outscored their control group peers on the Test of Economic 
Literacy by 2.60 items (effect size = 0.32). Student academic performance was also assessed using open- 
ended performance tasks that tested problem-solving abilities in short essays. On a composite score of 
these tasks, students in the intervention group outperformed those in the control group (effect size = 0.27, p 
< .05). 

One way to interpret the magnitude of these effects is to compare them with the overall progress 
that students make during an academic or calendar year. Hill et al. (2008) reported that 10th graders’ 
scores on norm-referenced tests in reading increase by 0.19 standard deviation units and in math by 0.14 
standard deviation units over a calendar year. Comparable growth information is not available for high 
school economics. One might imagine that not having taken any economics previously the growth of 
knowledge in economics would be greater than in these other subjects. However if growth in economics 
achievement is similar, the impact estimates are equivalent to at least one year of growth. 

Benefit to Teachers 

There were also statistically significant differences in favor of intervention group teachers on a 
measure of teacher satisfaction with the teaching methods and materials used to teach economics. The 
economics teachers who used the PBL approach were significantly more satisfied with the materials and 
methods than their peers who did not use the curriculum. The effect size was quite substantial (effect 
size=1.09, p < .01). 

No statistically significant difference was found between the intervention and control groups on 
teachers’ knowledge of economics, possibly due to a ceiling effect. Teachers averaged more than 90% 
correct on the pretest, which was the same as the student version of the test that was used. 

Finally, no significant difference was found in teachers’ pedagogical style with the survey 
measures used, based on a cut-off point using p value < .05. However, there was an effect size of .55, p < 
.07, even after multiple controls were in place, indicating a strong likelihood that PBL teachers were in fact 
more constructivist in their teaching practices as a result of using the curriculum. 

Conclusion 

The positive outcomes from this study are ground breaking in many ways. We are not aware of 
any other federally funded studies of this kind that examined PBL in a high school subject, or any K-12 for 
that matter. These findings add to the research base on use of PBL in economics, which had indicated 
promising impacts on student gains and teacher satisfaction. However, unlike previous studies, the design 
of this study allows for a causal interpretation of the students’ gains and greater generalizability. Educators 
may be looking for ways to strengthen their economics education programs; this study may provide useful 
information to curriculum specialists and teachers interested in alternative approaches for providing 
instruction in a required component of the high school curriculum in their state. 

However, in other respects this study only scratches the surface of our understanding of the impact 
of a PBL approach to teaching economics. A study like this only looks at “bottom line” differences 
attributable to the intervention. It does not address variations in the outcomes due to the quality of 
implementation or differences within teachers (e.g., who was more successful teaching economics than 
others using the curriculum and with which students, or how their practices in general and implementation 
of the curriculum differed). 

Observations of classrooms indicate that implementation the PBL units varied enormously 
depending on the individual teacher. It was not clear that professional development session impacted 
teachers’ use of the PBL materials compared to how they might have used the materials without 
professional development. 

Observations of control group teachers also indicated a wide range of practices, including some 
very effective and clever use of methods that resembled PBL at times, use of Socratic discussions and other 
seemingly useful approaches. At the end of the day, it is very hard to get an impact in a study like this. 
There were a lot of teachers in both groups who were extremely engaging and had a certain “spark” for 
economics that was conveyed to students, while in both groups there were teachers who lacked this spark. 



195 




(Personal communication with Neal Finkelstein, September 24, 2010) 

The differences in teaching practices were not statistically significant, suggesting the difficulty of 
capturing pedagogical differences. (Although self-reported practices almost showed a statistically 
significant difference (Effect size = .55. p < .07), there may be better ways to document differences in 
teaching quality both within treatment and control conditions. One such approach has been developed in 
large scales studies such as reported by the TeachScape project (Pea, et al., AERA 2010), which analyzes 
pedagogical practices using audio and video. It would be interesting to see a study of this curriculum that 
more carefully addressed pedagogical differences and how these impact learners. 

A great deal can still be learned by understanding how the supports offered to teachers, which 
included online supports, were used and can be used in the future to scale use of PBL. The intervention 
included some technology supports via video conferences, but these were not studied and evaluated. A key 
question to be addressed is how to support more effective use of PBL and it is likely that new technologies 
can play an important role (e.g., see Ravitz, 2010b). Perhaps there are ways to offer online professional 
development for teachers who obtain the curriculum or even ways to put the curriculum itself into an online 
format for use by teachers and students. (Note. Social Studies School Services is now publishing this 
curriculum without a professional development offering under the name of Project Based Economics. 
http://www.socialstudies.eom/c/product.html7reeord@TF43284). 

While the PBE curriculum appeared to be beneficial as compared to the approaches used by 
teachers in the control group, according data in the TEL handbook (Walstad & Rebeck, 2001) , both the 
treatment and contrast group in our study below the norm scores for country. This means that although the 
PBE group appears to have benefitted, there is still a long way to go in delivering high quality economics 
instruction in the participating states. It is possible that the characteristics of schools where the study took 
place are different from the schools in which TEL was normed. For example, California and Arizona may 
have larger than average number of language minorities and low SES students. We still have a lot to learn 
about how PBL works with diverse students, including language learners. It is unclear how much the 
intensity of language required for PBL or the assessments that were used may have influenced the study 
results, and whether the curriculum may have measurably helped students with some backgrounds more 
than others. Future analyses of the data will examine which types of students did well with the PBL 
economics curriculum, and how implementation can best support different kinds of learners. 

There is an inherent inability for this study, as it was designed, to untangle curriculum and 
pedagogy. The treatment group received not just a PBL approach to curriculum but a series of information 
session and materials designed to support their content knowledge and teaching during the curriculum. 

This makes drawing conclusions about the effectiveness of the PBL approach problematic; the difference in 
outcomes could be attributable to teacher learning and access to higher quality content, not just a new 
approach to teaching that content. We are not sure how you could separate these things, but perhaps the 
control group in future studies could receive resources designed at improving teacher content knowledge 
and ability to present that content, independent of the framing of this content by a PBL unit. 

There are only so many economics teachers per school, so it is important to consider how the 
lessons from the economics study might apply to other grades and subjects. This curriculum was 
developed over the course of several years and involved experts in instructional design and content areas. 
This development effort is clearly more substantial than most teachers would be able to undertake 
Flowever, in recent years, thousands of teachers have received professional development to design their 
own PBL units (tailored to their subjects and grade levels and students), and large libraries of projects have 
been developed using templates designed to scaffold planning, management and assessment of PBL (e.g., 
Buck Institute for Education, 2010; Williamson, 2008). The growth in the number of teachers who are 
trying to create well-designed PBL in an important development and there is very little research being 
conducted that might help people better understand how to effectively support their efforts. 

One area of research to investigate in the future includes better understanding the conditions 
required for effective use of PBL. The intervention for this study was quite extensive. It will be important 
to figure out what is required if more people are to teach this way in economics and other subjects. People 
who have developed “small ID” reforms (lessons, curriculum) have often suffered from lack of “big ID” 
contexts (systemic change) for their work. The blossoming of large-scale PBL initiatives brings forth a 
plethora of opportunities for graduate students, researchers and developers (e.g., to study performance 
assessments, group work, student-as-researcher, specific technology scaffolds, etc.). Many challenges still 
face PBL users and researchers, but there has been progress in designing schools to support PBL (e.g., 
Author, 2010; Expeditionary Learning Outward Bound, 1999; Pearlman, 2002), creating online scaffolds 
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for PBL use (e.g., Hmelo-Silver, Duncan & Chinn, 2007), administering large scale performance 
assessments (e.g., Silva, 2008), measuring youth development outcomes (Newell & Van Ryzin, 2009). 

Now we have the first randomized controlled study to support PBL’s effectiveness. 

After answering the question “does it work” in this large study, it is still critical to learn how PBL 
can work better and be made more accessible across schools and subjects. There are unprecedented 
opportunities for research and it is important for the field to be aware of these. With the release of this new 
study, interest in using PBL to support effective teaching and learning should grow. 
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How could we use fantasy for learning in educational games: the first attempt 
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Abstract: The purpose of this study was to explore fantasy meaning in learning technologies, especially an 
educational game setting. According to the literature and prior research base, fantasy has 2 main factors in the innate 
meaning. As a result of this research, the scope and depth of fantasy aroused from media imply what functions of 
fantasy in educational games are, how to use fantasy as a learning facilitator in learning technologies and design 
fantasy for learning on tailor-made purposes. For purpose of this study, the effects of fantasy in educational games 
on student academic achievements are explored. As a learning technology environment, the MMORPG (Massively 
Multiplayer Online Role Playing Game) Nori-school was used by the participants, consisting of third graders in 
elementary school. The operational definition of fantasy in educational games was set as an independent variable. 
Interest, intrinsic motivation, and storytelling as a narrative test were set as mediating variables, and academic 
achievements were chosen as a dependent variable. Interest, intrinsic motivation, and storytelling as a narrative test 
which are the mediating variables, affect the academic achievements. Results imply that learning technologies in 
conjunction with fantasy can be an effective way to increase students’ cognition toward academic achievements. 

Keywords: fantasy, fantasy and learning, communication in learning technology 

1. Introduction 

The self in post-traditional societies has been proposed as individual practice and narrating their identity 
(Hartley, 2005; Beck and Beck-Gernsheim, 2002). Personal media in post-traditional societies suggest a reflexive 
relation forming identity in social networks (Beck and Beck-Gernsheim, 2002). 

Human’s cognition as an agent of information processing can’t equal to the reality as phenomena. Shared 
cognition is not objective real since it connotes prejudice, distortion, or stereotype of their culture. Thus, some 
culture areas might call our collective cognitions as a fiction, their fantasy. However, we have shared our cognitions 
and that is able to communicate through inter-subjectivity, thus we are grounded on collective fantasy, culture code. 

Briefly, fantasy means a personal world in consciousness. According to research about memory (Eichenbaum, 
2002), forming identity is the prerequisite of building their constructs. Personal media could give people potentials 
or chances for constructing and identifying their identities by freely expressing and realizing in social networks or 
personal settings with using the features of media. For instance, Web 2.0 as a learning technology which can suggest 
learner participation and creativity and online identity formation is come into the spotlight. 

Fantasy as a concept of personal information processing might be meaningful for improving memory which can 
link academic achievements. Based on shared fantasy as cultural level, how can we use the individual memory 
which is formation of complexity between cognition and emotion for improving achievements? 

2. Background 

Educational or serious games might suggest the most fantastic media features as a learning technology. Most 
experimental research on the adoption of educational games on fantasy and learning has shown that digital games as 
a learning environment are presented in a fantasy context that is of interest from fantastic features and leads to 
increases in both student’s interest and learning (Parker and Lepper, 1992; Cordova, 1993; Cordova & Lepper, 
1996). High levels of interest arouse and maintain an intrinsic motivation for learning (Hidi, 2000). Interest is 
associated with deeper narrative of education and it may be the primary affective component in engagement 
(Consedine et al., 2004). Narrative is a way of resolving some fundamental antagonism by rearranging into a 
temporal succession (Zizek, 1997). Thus, fantasy contents which are based on need have been analyzed by using 
narrative structure (Scott, 2001). 

Learning is associated with play benefits related to factors such as increased motivation (Rieber, 2001), and 
computer games are reported to increase motivation of learning as well (Asagari & Kaufman, 2004). The fantastic 
features may increase intrinsic motivation in individuals through satisfying their needs in its field (Asagari & 
Kaufman, 2004). In addition, narrative is a potentially effective tool for exploring the structure and process of 
“meaning making,” as evidenced by psychologists adopting techniques from narrative analysis (Bruner, 1990). 
Narrative learning could offer more motivational and cognitive benefits than traditional learning experiences. 
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2.1. Fantasy 

Malone and Lepper (1987) define fantasy as an environment that “evokes mental images of physical or social 
situations not actually present.” Psychiatric researchers define fantasy as a “defense mechanism” for the fulfillment 
of wishes and the resolution of conflict (Caughey, 1984; Hook, 1979). According to Hume (1984), fantasy is any 
departure from consensus reality, an impulse native to literature and manifested in innumerable variations, from 
monster to metaphor. 

As indicated by Tolkien (1961), the reason we use “fantasy” is that it combines with its older and higher use as 
an equivalent of imagination the derived notions of unreality, of freedom from the domination of observed “fact”, in 
short of the fantastic. 

To synthesize the above information, the operational definition of fantasy is as follows. Fantasy is an 
imaginative personal mental process or operation for substitute behavior that intends to fulfill the needs of people 
(Table 1). 



Imaginative personal mental process 


Imagination 


Substitute behavior 


Memory intervention 



Table 1: Components separated by concept of fantasy 



In the sense of Philosophy and Sociology, fantasy patterns are separated like Figure 1. According to this 
stereotype, the concept of fantasy’s degree in educational games can be applied as below. 



Shared fantasy 


- 


the scope 
accepted 


- 


Ego-centric fantasy 



Figure 1: Fantasy patterns in contents 



2.1.1. Shared fantasy 

Games with no fantasies involve only abstract symbols (Malone, 1980). This pattern of fantasy is to experience 
mental operation or process based on the real. The real is represented by some specific culture of the related people 
on the game. The real of gamer and designer is made up of their culture and cognition formed by socialization. Not 
only do objects or storyline biased in game, by means of the absolutism perspective, but also they are containing 
elements of the culture system so called common illusion as an agent of sharing fantasy like ego, moral laws, 
religion, economy, science, and arts. Common illusion can be separated by a scope such as the world, a country, or a 
group. The wider accepted scope is the more stuck reality. Representations reflected by designer's view and 
interactions of massive multiplayer game, which are a fusion of their common illusion as a subculture. Objects, 
storyline, and interactions which we have felt authentic or realistic are cultural results. As the inherent properties of 
community through interactions, personal mental operation and process might be controlled by fantasy of reality as 
common illusion. 

Thus, a game which has attributes of social grouping tends to arouse collective fantasy which lacks features of 
egocentric or autistic qualities that Freud and Piaget discuss as characteristic of fantasy states(Fine, 2002). Since 
human’s needs to fulfill are focused on communication to achieve goals in massive multiplayer, objects or storyline 
could be coated just as materials. That is, a mechanism of fantasy as reality turns up more frequently than personal 
role-playing game does. 

2.2.2 Ego-centric fantasy 

This pattern is to experience imaginative mental operation or process so called defense mechanism for the 
fulfillment of individual unconscious wishes and the resolution of conflict (Caughey, 1984). Wishes and needs take 
one’s mental ground from experiences distinguished. A feeling of satisfaction could begin to escape the present 
states. It can provide opportunities to undergo achievements couldn't be gained in actual land. Human meet their 
needs through internal dialogues, and then they feel catharsis. This is the reason why a game could induce 
engrossment and promote identification with the figures. 

2.2. Fantasy in learning, especially educational games 

Children have a unique opportunity to explore their own emotional arousal (Fein, 1987), and an opportunity to 
experiment with possible interactions and relationships among humans (Leslie, 1987) through pretense or fantasy 
play. Fantasy play fosters children’s cognitive and language skills (Cassell & Ryokai, 2001). In fantasy play, 
children practice their ability to represent objects and feelings with something that stands for them (Cassell & 
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Ryokai, 2001). Peer’s stories and imaginary creations serve as new suggestions for children to enact and tell their 
story creatively and freely (Baker-Semiett & Rogoff, 1992). In sum, fantasy might function to fun and information 
processing mechanism (Table 2). 

Fantasy 



/ \ 

intprpct I I information 

imereM | | processing 

Fig 2: Fantasy loop in learning 

As mentioned by Tolkien (1961), fantasy is the mental complexity as an equivalent of personal imagination and 
reality ranged by a person’s life scope. As a unique mental complexity adopting personal imagination, fantasy has a 
differentiating feature from usual information processing. Thus, fantasy can be separated in a similar way to mental 
information processes and, may also follow perception through sensory functioning, representation and action by 
motor functioning of brain. These stages are the conjunctive points of processing adjustment between inner 
consistency and new stimuli from learners’ outside as well. Therefore, fantasy in an educational game might be 
observed in a view of pathway of sensory stimuli in physiology (Fig 3): response, narrative, and creation, and it is 
reflected to Table 2. 

First, in the moment learners see or interact with fantastic features, they can experience their fantasy as 
response. It is triggered by fantastic features made by media characteristics, which are so called “bells and whistles”. 
Sometimes it is used as a feedback loop to gamers for supplying rewards, when a gamer accomplishes the mission, 
such as sounds and messages of congratulations, shining images, or three-dimensional (3D) actions. Second, when 
they start to follow the scenario of the game, gamers’ narratives, however, are different. Gamers concentrate on 
game environments first, and build their own meaning by inner consistency against new stimuli second. That is, the 
spacing between the lines of concepts is fulfilled by personal narrative built by gamers’ own constructs. Third, if 
they change or create something in a game, they can participate in the game scenario as a creator. In Second Life, 
making an HUD (Heads up Display), transforming objects, and generating and shifting appearance are adaptive 
instances. Avatars, fantastic features, and many functions for controlling their actions in a game environment 
facilitate learner’s fantasy, and learners can experience their fantasy by interacting with their past or prior experience 
or constructs. Table 2 shows a summary of this concept. 




Fig 3: Pathway of sensory stimuli in physiology 



Information processing 


Fantasy type 


Fantastic features in games 


Sensory functioning 


response 


Bells and whistles such as a fantastic 
sounds, images, or actions 


Representation 


narrative 


Inner construction through interaction 
user’s inner construct with symbols 


Motor functioning 


creative 


Freely expressing using game control 
functions their thoughts or themselves 



Table 2: Comparison of fantasy type based on information processing 



For expediting fantasy, there are some environmental conditions. The first condition is a pennissive atmosphere 
of free thought in which their narratives are respected and their storyline of mind can be reflected. This has a role of 
activating chances of memory intervention. The second is stimulating environment for imagination such as sensory 
richness or suggesting refreshing images. The more various and new sensory channels experienced by people, the 
deeper and wider their imagination scope becomes. The third is an advantageous environment for having projective 
identity. If humans’ needs can be fulfilled in some situations, their immersion is deeper. Immersion is the necessary 
condition for evoking fantasy, and the relationship of projective identity and immersion is clinical. Projective 
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identity is formed by mirror neuron mechanisms of the brain. According to Meltzoff and Prinz’s research (2002), 
mirror neurons might facilitate preliminary motor neuron simulation, priming, programming, and rehearsing, and 
this process enhances their ability to read the minds of others. Seeing characters in a game activates mirror neurons 
of gamers’ brain, and it is possible to auto shape projective identity about their seized character or as a member of a 
virtual community. 

Games suggest narrative structure by a pre-determined storyline and give chances for reflecting gamers’ 
narrative. For example, some games do not include an ending scenario. Instead, gamers make their own world, and 
this becomes the result of the game. The characteristics of the game, such as 3D effects and virtual adventures, 
supply the fantastic features to gamers’ group as a community, thus they can feel the pleasure of triumph from this 
world, added to the winning experience for each stage and can share it with other game members. They could realize 
their needs and be immersed in the game situation and then, their world is added to their constructs as virtual 
experiences. Virtual experiences act on their cognition as a mental complexity between the real and the imaginary. 

3. Relationships among variables 

Interest has been defined as the most basic and ubiquitous of universal motivating emotions for humans (Bye et 
al„ 2007; Izard, 1997, 1993). The proximal motivator for persistence and subsequent engagement, especially for 
activities that take place over the long term, is the degree to which one experiences interest and enjoyment in a 
particular task (Sansone & Smith, 2000). Particularly, intrinsic motivation has been described as the source of 
strength from individual interest (Alexander et ah, 1997). 

Csikszenmihalyi (1997) has described the auto telic or intrinsically motivated person as one who pays attention 
to things for their own sake without expecting an immediate return, is capable of sustained interest without 
recognition or with little support, and becomes caught up in the feedback loop between learning, interest, and 
enjoyment. An individual can enter the flow state by new skills, challenges, or new cognitive layers to existing 
interests built on the foundation of prior experience (Bye et ah, 2007). 

Tuner (1981) suggests that narrative is an experiential knowledge. The pattern of individual narrative can be 
easily observed by storytelling (Cassell & Ryokai, 2001). Narrative effects are included more or less unconsciously 
in most everyday storytelling; tellers explore and convey knowledge and experience. At the cultural level, personal 
narrative gives cohesion to shared beliefs and acknowledged values, so they are even more powerful when we tell 
the stories ourselves (Cassell & Ryokai, 2001). Especially, sociolinguistic discussions about narrative start with 
Labov and Waletsky (1967). The theory of natural narrative that emerged, focused on oral narratives of personal 
experience, describes a linguistic phenomenon that is both structured and detailed. Labovian Narrative Theory (LNT) 
makes clear that its discussion of narrative describes a very specific type of single-person narratives of personal 
experience. It notes that such narratives overwhelmingly tend to favor a very specific structure and organization, 
starting with the speaker signaling intent to begin a narrative, then a brief outline signaling the “point” of the story 
(Abstract) followed by context and setting (Orientation) within which the personal experience is embedded 
(Complication). The actions or events in the narrative then lead to some climax or effect (Resolution), which the 
narrator interprets or presents in some manner (Evaluation) reflecting her memories and stance toward the 
experience. Finally there is a conclusion or concluding statement (Coda), marking the end of the story. 

Fantasy is, in accordance with learning principle viewpoint, the concept of respecting the maximization of 
students’ constructs as a mental process which is aroused by interaction with gamers and context. After that, 
although learners have a real physical existence with their own identity of thinking because they have the experience 
as a main knowledge agent of thoughts, they can be interested in the suggested context. That is, the rich sensory and 
motor environment and elements of games can facilitate learners’ free and infinite thinking, which leads to increased 
intrinsic motivation. According to accepted Carnival pedagogy (McCathy & Dimitriadis, 2004; Rabkin & Redmond, 
2006) which emphasis on daily life of learning, voluntary and active participation and enjoying oneself, however, 
positive effects about learning context such as fun, interest, and expectation can link to achieve academic grades 
through opportunities to experience ordering and withdrawing their world which is fantasy. For academic 
achievement, learners’ private fantasy needs to order based on common fantasy in their culture like using a method 
of storytelling for expressing their narrative. Especially, storytelling has a role of communication in their community, 
thus, that would be used in a representative way for consultation of their narrative with others. 

In summary, the current study aims to address the following research questions: 

1 . Are there direct significant effects of fantasy in educational games on achievements in learning? 

2. Are there indirect significant effects of fantasy in educational games via interest, intrinsic motivation, and 
storytelling on achievements in learning? 

In answering these research questions, the appropriate methodological tools are those of path analysis. 



203 




4. Research Method 



4.1. Participants 

The subjects used for this experiment were drawn from an elementary school located in Daejeon, South Korea. 
The participants were composed of 120 third grade students who had never played the game Nori-school. Their ages 
ranged from 9 to 10 years. 

4.2. Instruments 

Measuring the variables is composed of fantasy in educational games, interest, intrinsic motivation, and 
storytelling. The on-line game. There are many genres of computer games including action games, adventure 
games, strategy games, and role playing games. In this study we used a commercially available MMORPG known 
as Nori-school. Nori-school is a popular MMORPG in Korea that is set in the English learning context of a wizard 
town. Players receive a variety of quests from Non-Player Characters (NPCs) and have to solve these quests to 
become a powerful avatar. In the English learning scenario, Nori-school allows players to interact with objects in 
hospitals, houses, and factories. The learning goal of Nori-school is to memorize, understand, and apply expressions 
of English. Fantasy in educational games. This is a questionnaire of 18 items (Likert scale) modified 
appropriately based on concepts of imagination, memory intervention, and identification for educational game 
setting and elementary school students to be based on the creative experiences questionnaire (CEQ) developed by 
Herald Merckelbach et al. (2001). Cronbach’s Alpha value of the inventory was .83. Interest. This consists of 
15 items based on Schiefele’s (1991) theory of interest. In addition, five items of The Differential Emotions Scale 
-A (DES; Izard, Libero, Putnam, & Haynes, 1993) are used for measuring more persistent interest. Cronbach’s 
Alpha value of the inventory was .73. Intrinsic motivation. The intrinsic motivation inventory (WPI: Work 

Preference Inventory) developed by Amabile, et al. (1994) was selected and adapted by the author. The scale 
consists of 10 items. Cronbach’s Alpha value of the inventory was .78. Storytelling (narrative test). When 
learners finish the game Nor-ischool, they wrote reflective diaries focused on frequently used expressions as 
learning contents in Nori-school. It was graded by a 3-point scale how much they wrote expressions used in Nori- 
school depending on their own words. Academic achievements test. An achievement test developed by the 
author was administered to participants. The achievement test was used to measure student’s achievement about 
memory or understanding of words. The test consists of words of 50 questions suggested by the second stage of 
Nori-school. Cronbach’s Alpha value of the inventory was .82. 

4.3. Procedure 

First, the teacher explained how to play the game. Then, the participants played Nori-school until they all 
reached the first level. This was done to ensure that players have the same gaming ability. They played the online 
game, Nori-school, about 80 minutes per day (sum of classroom and homework), three times a week, for four weeks. 
Then, interest, intrinsic motivation, narrative, fantasy proneness, and the academic achievement were administered. 
As we intended to examine students’ academic achievement by fantasy, game scores were excluded in this 
experiment. 



5. Data analysis 

Quantitative data were collected and analyzed using SPSS 15.0 for Windows and AMOS 7.0. The dependent 
variable is academic achievements. The independent variable in this study is fantasy in educational game. The 
mediating variables are interest, intrinsic motivation, and storytelling. Path analysis was conducted to examine the 
correlations and causal relations between the independent and dependent variables. Developed by Sewell Wright in 
the 1930s, path analysis is usually used to understand the relationships among variables. In path analysis, covariance 
or correlation coefficient is used to do causal analysis. It can be used to understand direct and indirect effects as well 
as quasi-effects that are hard to observe in multiple regression analysis. This means of analysis is used to find the 
regression coefficient of the linear structure formula among the variables set by the investigator. 

In addition, narrative analysis was conducted to find fantasy contents from reflective dairies, based on the 
scope accepted. The tendency between shared fantasy and ego-centric fantasy was counted. 

6. Results (1) 

6.1. Analysis of correlations between variables 

A correlation analysis was performed to determine the relationship among all the variables. Table 3 shows the 
correlation coefficient among variables. Fantasy in educational games appears to be significantly related to the 
interest, intrinsic motivation, narrative, and academic achievements. Narrative and intrinsic motivation are 
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significantly related to the achievements in learning. Interest, intrinsic motivation, and narrative have a statistically 
significant relation to the achievements in learning (Table 3). 



Table 3: 

** Correlation 
the 0.01 level 

6.2. Path 





Fantasy 


Interest 


Intrinsic 

motivation 


Storytelling 


Academic 

achievements 


Fantasy 


1 










Interest 


.50** 


1 








Intrinsic 

motivation 


.60** 


.51** 


1 






Storytelling 


.67** 


.50** 


.76** 


1 




Academic 

achievements 


.53** 


4g** 




74 ** 


1 



Correlations, 
is significant at 
( 2 -tailed) 

analysis 



between the variables 

The research model was established based on the idea that fantasy in educational games has an effect on 
interest, intrinsic motivation, and narrative, and that interest, intrinsic motivation, and narrative have an effect on the 
achievements in learning. The path model is presented in Figure 4. In assessing the model, the author relied on 
several standard fit indices to examine the overall model fit: GFI, CFI was calculated as in Table 4. The chi-square 
was 27.216, p (probability level) = .004, the chi-square/degrees of freedom ratio (Q) was 2.474, the GFI = .920, the 
CFI was .964 indicating that our structural model was a satisfactory fit for the data. 
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CFI 
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.942 


.932 



Table 4: Summary of fit indices for measurement models 





Paths 




Estimate 


S.E. 


C.R. 


P 


Label 


Intrinsic motivation 




fantasy 


1.793 


.304 


5.895 


*** 


accepted 


Interest 




fantasy 


3.151 


.589 


5.351 


*** 


accepted 


Storytelling 




fantasy 


.306 


.052 


5.894 


*** 


accepted 


Achievement 




Interest 


.118 


.064 


1.831 


.067 


dismissed 


Achievement 




Intrinsic motivation 


1.126 


.290 


3.875 


*** 


accepted 


Achievement 




Storytelling 


5.539 


1.694 


3.269 


.001 


accepted 


Achievement 




fantasy 


-1.486 


1.213 


-1.225 


.221 


dismissed 



Table 5: Estimate of path coefficient, *p<0.05 **p<0.01 ***p<0.001 



1.49 




Figure 4: The path analysis model 

Results of the path analysis are shown in Table 5 and Figure 4. The fantasy in educational games does not have 
a significant effect on the achievement in learning and is therefore dismissed. The fantasy in educational games has 
a significant effect on the interest, intrinsic motivation, and narrative (P=3.15, 1.79, 0.31, p<.001). The intrinsic 
motivation and narrative have a positive and significant effect on the achievement (P = 1.13, p< .001, P= 5.54, p 
< .001). But interest does not have a significant effect on the achievement and is therefore dismissed. The fantasy in 
educational games partly has a significant effect on the achievement in learning via the interest, intrinsic motivation, 
and narrative. As predicted in research problem 2, the fantasy in educational games had positive effects on the 
achievements in learning (Fig. 4), when it has mediate variables that are the interest, intrinsic motivation, and 
narrative. 
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7. Results (2) 



The purpose is to see how respondents in interviews impose order on the flow of experience to make sense of 
events and actions in lives. The importance of narrative analysis is how narrative is organized. Grounded the concept 
of fantasy contents, narrative analysis was conducted to select between shared fantasy and ego-centric fantasy. 
Shared fantasy has a tendency of the community values which seek to external needs desiring to be recognized. Ego- 
centric fantasy, however, is focused on internal needs relatively. Used game, Nori-school is Massively Multiplayer 
Online Role Playing Game. This game environment is able to communicate or establish some strategies to achieving 
game goals, at the same time individual playing is not unnaturally. Fantasy contents were investigated, with 
grouping to massively multiplayer (64) or individual player (56). Conducting narrative analysis, Table 6 is the result 
of counting two groups. 



Needs type 


Needs content 


Massively 

multiplayer 


Individual 

player 


External 

needs 


Predominance 


47(73%) 


24(43%) 


Control others 


57(89%) 


16(25%) 


Game goal oriented 


59(92%) 


32(50%) 


| *Rcal community rule based 


62(97%) 


25(39%) 


Internal 

needs 


Individual representation 


46(71%) 


51(91%) 


Self goal oriented 


28(44%) 


50(89%) 


Out of rules in real 


11(17%) 


45(80%) 



Table 6: Results of narrative analysis 



As a result of narrative analysis, in case of massively multiplayer had a tendency of distinctly seeking to game 
goal. Since they participated in the game for breaking game missions, they communicated by common language 
adopted in real community. Their social activities with cognizing other people might be conducted in the game 
environment. Massively multiplayer felt a sense of belongings in community more densely because of group goals. 
They could share their fantasy which is converted collective fantasy, and it might rearrange game goals fitted in their 
world. 

In contrast, players participated in the game individually were defined to their inner worlds relatively, because 
they didn’t need to communicate other players. They just achieved personal game goals until wanting to quit the 
game. Thus, the game might act as an environment concentrating on their projective presence. 

8. Conclusion and implications 

This study sought to investigate the effects of fantasy in educational games on academic achievements. Games 
facilitate learner’s fantasy through their media characteristics, given chances in virtual communities and game 
structures, and then educational games could act as a trigger to form learner’s narrative toward learning. Research 
problem 1 suggested, however, is dismissed, because fantasy questionnaires is examined to focus ego-centric. Since 
each character of a MMORPG has its own role and skills, and players gamed in groups are about half in participants, 
the possibilities of forming personal narrative structure could be restricted as well. 

Considering the path analysis model, fantasy has the strongest significance on interest. Following Flume’s 
interpretation (1984), “Tolkien meant the refreshing effect of defamiliarization, the newness available to us only 
after we have freed ourselves from our sense of possessing the familiar (p. 16).” In contents of fantasy, refreshing 
effect functions as ventilation of stimuli in intelligence and affect. Fantasy is a very strong factor for arousing fun 
and interest, thus, many students may show symptoms of game addiction. The second strongest variable affected by 
fantasy is intrinsic motivation. One of the distinguishing characteristics of intrinsic motivation from those of other 
motivations is “interest” about something endogenous. Interest is the strongest factor of fantasy in educational 
games, but it did not show a significant effect on academic achievements. Intrinsic motivation about learning 
contents, however, had a significant effect on academic achievements, thus, we could identify the importance of 
transitioning from interest to intrinsic motivation. That is, interest can just serve as a trigger to rely on media 
characteristics. 

According to a result of path analysis, narrative is the strongest variable affecting achievements in learning. If 
learners begin to own their narrative about learning context regardless of the format supplied, they can achieve the 
goal of learning more effectively. Thus, it is needed to develop how to design the fantastic features for forming more 
efficiently and effectively personal narrative as information processing about learning contents and how to suggest 



206 





the chance for withdrawing from their narrative for academic achievements. 

Conclusions which can be drawn from this study are as follows. First, fantasy in educational games needs 
conjunction strategies such as intrinsic motivation and using narrative for increasing students’ academic 
achievements. Therefore, it is desirable for designers to find factors of fantasy in educational aspects, map them (the 
fantastic features) to their curriculum objectives, and apply benefits of fantasy for players in order to secure learning 
effectiveness. Second, designers of educational games need to focus on adopting strategies for forming and eliciting 
learner’s narrative about learning from their fantasy such as storytelling or creative activities. Fantasy has a property 
of narrative, but the complexity of fantasy can interfere in withdrawing their narrative for achieving learning goals. 
Personal stories are not merely a way of telling someone (or oneself) about one’s life; they are the means by which 
identities may be fashioned (Rosenwald & Ochberg, 1992, p.l). Narrative has a strong significant effect on 
academic achievements, thus, designers of educational games have to consider strategies for systemically ordering 
learner’s thought from their fantasy to common fantasy. Storytelling elicits their experience or constructs including 
present learner’s situations and the eliciting could serve as another experience through ordering narrative as a 
complexity of fantasy. In this research, in that massively multiplayer, they had a chance of sharing their world 
through their common real rule and language, they experienced storytelling in a group. Flowever, their storytelling is 
just their collective fantasy in their group, thus it needs to be differentiated from the meaning of ordering gamers’ 
narrative. Divergent thinking can be extended in educational games as sensory richness environments, but it is 
meaningful on academic achievements when it is linked to opportunities for learner’s expression, which can gain 
experiences of reconstruction. This study suggests that designers of educational games need to set strategies which 
facilitate the building and extracting of learner’s narrative as a mental complexity, since fantasy in educational 
games has no connection with learning context directly, but fantasy can be used with mediating variables for 
academic achievements. If this is done, then educational games might be more accepted as an effective teaching and 
learning tool in formal educational settings. 
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Abstract 



The application and analysis of instruction in an online environment is critical to the success of distance learning. 
Whether a web page, blog, or content delivered by a learning management system, the instructional design is key to 
increasing the level of interactivity among participants. Using an Action Research model, the Interactive Learning 
Taxonomy (ILT) has been integrated throughout an online course for K-12 teachers. This study shows examples of 
the taxonomy, of interactive learning elements, and of the continuous improvement of learning outcomes in an 
online course. 



Review 

Online learning in K-12 schools, in community colleges, and in higher education is the choice method of 
delivery. As traditional K-12 educators move to the role of instructional designers of online curriculum, it is 
important to recognize that what works in a face-to-face classroom filled with text and textbooks does not always 
work in a web environment. Online learning is not flat, is not text heavy, or a replication of the textbook. Good 
online content invites, motivates, and engages the learner to interact with the online content. The term engaged is 
used here to emphasize that the participants interact with each other around substantive issues (Xin, & Feenberg, 
2006). In addition, Mitsuhara, Kanenishi, & Yano (2006) coined the phrase multiperspective thinking in reference 
to learners needs to think about online content from many perspectives beyond their individual interests. 

The term, “interactive learning” is described as the decisive measure of engagement in an online or in an 
on-ground class. As a dynamic relationship between teacher and students, between students and resources, and 
among students, interactive learning encompasses all content areas and all dimensions of learning. Steinaker and 
Leavitt (2008) designed the Interactive Learning Taxonomy (ILT) as one of 17 taxonomies educators could employ 
while engaged in the act of teaching and learning. 

The role and function of computer enhanced learning as well as the level of interactivity for the teacher and 
learner has long been studied in the literature. “Computer based instruction provides greater potential for truly 
interactive instruction than any mediated teaching device. . .excluding the human tutor (Jonassen, 1988:97, in Sims 
2003). To frame how digital content becomes “interactive” the ILT and its inquiry needs to include many of the 
contemporary digital construction tools such as blogs, wikis, learning and content management systems and other 
Web 2.0 tools that allow for the easy construction of digital, multimedia enhanced instruction. 

The purpose of this paper is to report ongoing action research conducted with K-12 educators in a Masters 
degree program, which specializes in educational technology with a focus on teaching online. Action Research is 
defined as the process of "any systematic inquiry conducted by teachers in a teaching learning environment, to 
gather information about the ways in which their particular school operates, the teachers teach, and the students 
learn” (Gay, Mills & Airasian, 2006, p. 595). An Action Research model has been applied to examine if the ILT 
could be effectively utilized in an online environment. This study also seeks to exemplify the ways in which each 
step in the taxonomy is made intrinsic to the online learning process. Following Davis and DunhilTs (2008) model 
of Learning Study, lesson preparation is preceded by an attempt to identify variation in ways of understanding a 
phenomenon that is the focus of the lesson. The intention of this activity is to help students to revise their 
professional knowledge, their theories of learning and teaching, in the light of their experience of practice. 

Through this ongoing research, we continue to collect data on the application of the ILT as a practical tool 
for the construction and analysis of online content and online instruction. Although this research reports the role of a 
student constructed lesson using a blog tool, there are other learning and content management systems that will be 
studied in future research. 

Theoretical Construct of Interactive Learning 

The ILT was first published in Steinaker & Leavitt (2008). The five categories of the ILT are: invitation, 
involvement, investigation, insight, and implementation. At the first stage of the taxonomy, activities in the 
invitation category are designed to engage students. Students need to become engaged in terms of content and 
process and then become aware of the short-term content as well as long-term goals and outcomes. Activities in this 
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category involve interactive dialogue as well as developing a sense of readiness to learn. Involvement with the 
content follows as a second category where it is essential for students (as navigators through the content) to develop 
new levels of interaction. Involvement is the step when connections are more firmly established with the content. At 
this phase, the student explores ideas, issues, and materials. 

As relationships grow, so does the next category of investigation. Here the learner begins to work at the 
application and analysis levels of cognition (Bloom, 1956). In this category, discourse and discussion are central to 
investigation. Group interaction among peers online is critically important. As a key component of the ILT, 
interrogative interaction with the teacher is pivotal in this investigative stage. It is hypothesized that insight develops 
when ideas and concepts come together and the process is internalized. Students demonstrate that they have 
achieved insight through analysis of goals and through expected outcomes. The process of developing insight from 
an interactive lesson can be both a culmination and a challenge for the student. The challenge becomes how to use 
what they have learned within the real world of their classroom or at their workplace. Implementation is the fifth and 
final category. Implementation is the dissemination and sharing of the changes that have taken place during this 
process of thinking, learning, and teaching. In this category of the ILT, students assume responsibility for what they 
have learned. They demonstrate their roles as influencer and disseminator. These are the categories of the ILT and 
the proposed process through which students will experience when they go through the taxonomy. 

Background of Study 

EDT 610 Teaching Online is the second 
class in obtaining a Masters of Arts (MA) degree 
with a specialization in Educational Technology. 

The specialization consists of four focused 
technology integration classes as a supplement to 
the (MAT) offered by the School of Education at 
National University. This specialization is 
designed to assist practicing teachers to enhance 
their teaching skills and to develop knowledge and 
skills for using technology in an educational 
setting. EDT 610 is a comprehensive course that 
covers principles and strategies for conducting 
online instruction in a variety of online teaching 
environments including hybrid and blended 
instruction. In this class, students survey theories and explore the application of online learning and teaching for 
youth and adults. During this class, students are engaged in both asynchronous and synchronous discussion groups. 

As the culminating project for this class, students were asked to create an interactive, online lesson using a 
blog format. A blog is "web log", an online lesson designed by the teacher to teach a concept or concepts in a 
specific subject and grade level. Blogs are widely used in K-12 education as a format for posting content and for 
communicating thoughts and ideas. Much of the "blogosphere" outside of education is used for logging personal 
comments and opinions. For the purpose of this class, students created a blog in EduBlog. 



(a) (j 



Educational Technology 610 

Technology-Supported Global Learning 



EDT 610 Teaching 



Demographics and Description 

In March 2009, the Interactive Learning Taxonomy (ILT) was integrated and applied in the EDT 610 class 
as a focused assignment for the analysis of an online lesson. Formerly, the class did not contain a model for building 
and assessing an online lesson. This pilot class included 13 students who were enrolled in the course: 7 males and 6 
females. For this initial study, 12 students completed the application and analysis of the ILT. Since March 2009, 2 
additional classes were added to the study, March 2010 (n =19) and July 2010 (n =21). 
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Process for Analysis and Application of the Taxonomy 

Students in EDT610 were asked to review a blog created by a former student, C. Lee (2009). An ILT 
template describing each stage was to evaluate the lesson/blog. Responses had to be three or more sentences in 
length. In addition, students were asked to rate the lesson according to criteria to determine the extent the blog 
followed the ILT criteria for interactivity and to rate at which level the National Educational Technology Standards 
(NETS) were mastered. These standards were developed by the International Society for Technology in Education 
(ISTE): Standard #1 - Facilitates and Inspires Student Learning and Creativity (NETS, 2008). The overall 
evaluation of the lesson/blog was rated using a 4-point Likert scale. 

Evaluation of Online Lesson/Blog with Application of the ILT (Sistek-Chandler, Amber, Steinaker, & Tolbert, 
2008) 



1 . Invitation - How has the instructor motivated and invited the learner? 

2. Involvement - How does the lesson involve the learner with the content? 

3. Investigation - Does the content of the lesson use an investigative strategy for learning the content? 

4. Insight - What insight does the student gain from being involved in the lesson? 

5. Implementation - How does the lesson and learning encourage the learner to implement or apply the 
content? 

Does this lesson address the ISTE/NETS Standard #1? 

1 -Doesn’t Address 2-Somewhat 3 -Mostly Addresses 4- Completely Addresses 
Study Results 

In the initial pilot study, twelve students completed the review of the lesson/blog and provided feedback 
that enabled the researchers to gauge their understanding of the taxonomy and consider it for future analysis. 

Analysis Questions: 

1 . How has the instructor motivated and invited the 
learner? 

Responses to the Invitation stage indicate 
the students perceive the site as meeting this stage of 
the taxonomy. Students mentioned such elements as 
“eye-catching and topic appropriate graphics; simple, 
attractive, well-organized format; page is easy to 
follow; and material is introduced with examples.” 

2. How does the lesson involve the learner with the 
content? 



Responses to the Involvement stage note 
that learners using the blog are required to interact 
with the content. Comments “builds on previous knowledge; good use of a variety of websites; provided California 
standard; rich print environment; allows students to interact with technology; and requires students to create a flow 
map and answer questions.” 

3. Does the content of the lesson use an investigative strategy for learning the content? 
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The action words used by respondents indicate their view that students are involved in investigative 
activities such as “students explore their surroundings and relate material to previous knowledge; students list 
adaptation of living organism; applying concepts; responding to posts; choosing an environment; and visually 
expressing results.” This indicates students are challenged to use higher order thinking skills. 

4. What insight does the student gain from being involved in the lesson? 

The Insight stage was seen as a strong element of the class blog. Comments from this area detailed 
activities that would encourage students to apply the knowledge in practical ways: “Students respond to posts and 
use critical thinking in their descriptions; students choose an environment and adaptations; students learn to visually 
express what they’ve learned; reinforcement occurs through use of external websites and activities; learning by 
reading other posts; encouraged to apply content; website encouraged real-life application; lesson draws 
comparisons to stages students will go through; students create a flowchart, and use knowledge to solve crimes; 
students provide their own examples through blog postings. 

5. How does this lesson and learning encourage the learner to implement or apply the content? 

Students documented activities that lead to implementation of the content: “respond to blog postings; 
answer questions based on observations; students choose an environment and remark on adaptations; students use 
knowledge to solve crimes; websites encourage interaction and real life application.” Each of these elements 
encourages learner interaction with the content. 

The EDT610 students were also asked to rate how well the instructional blog addressed ISTE/NETS 
Standard #1: Facilitates and Inspires Student Learning and Creativity (NETS for Teachers, 2008). A 4-point, Likert 
scale was used by the students to evaluate the level at which the lesson complied with or met the ISTE standard: 1 - 
Doesn’t Address; 2 - Somewhat Addresses; 3 - Mostly Addresses; and 4 - Completely Addresses. 

In the July 2009 study, the mean of the 12 responses was 3.83. Seventy-five percent of respondents (n=9) 
rated the blog as 4; 17% (n=2) rated the blog 3.5. For the March 2010 section the mean score was 3.66 ( n= 19) and 
for the July 2010 section, 3.7 (n=20). 

In July 2010, many students readily adopted this analysis of an online lesson and have also applied 
reflection and analysis to the overall process. In the dissertation research conducted by Ma, Lai, Williams, Prejean & 
Ford (2008) teachers who engaged in online journals often lacked articulate and reflective attributes of meaningful 
learning. Reingold, Rimor, & Kalay, (2008) emphasize that learning does not take place without reflection and 
engagement of metacognitive processes. One student in particular has applied this metacognitive statement in his 
work, suggesting we add another category to the taxonomy as an aid for his personal construction of the lesson. 

Dear Professor, Given the form [template] that 1 received. . .1 am hesitant to suggest adding another "I" to 
the list without first consulting you. Given the research that is stamped all over. . .the course's textbooks, 
might there be a place for an "Interactivity" category in the taxonomy? Overall, I believe the model [ILT] 
is a useful checklist, but will include the interactivity in my own evaluations to insure that I create the most 
powerful learning tools that I can (Student, EDT 610, July 2010). 

Through continuous observation and analysis, we have noticed another theme. Students have also engaged 
in the reflective process of comparing the model (C. Lee’s blog) to their own blogs. 

Overall it is a great blog. I see many weaknesses in my own [blog that I have created] that I am not sure I 
will be able to easily fix. A well-thought out lesson will take hours to complete, refine and edit, but once 
the final product has been produced, it can be replicated year after year with ease (Student, EDT 610, July 
2010 ). 

Overall Findings 

The ILT, its structure and categories were generally easy to apply, initiating thoughtful responses to the 
prompts. All of the five categories were responded to in detail. The Insight stage was seen as a strong element of the 
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class blog. The principles of the ILT can be used to increase interaction and involvement in an online course. The 
ILT can assist the teachers in guiding the construction of content to be more interactive. As a tool for analysis and 
evaluation of an online, interactive lesson, the ILT proved to be extremely positive and useful; students easily 
embraced the taxonomy. Often students engage in the practice of restating factual information however in this case, 
the ILT fosters higher order thinking through application and analysis. Overall, the results indicate the ILT is a valid 
tool for analysis of interactivity for practicing teachers. 

The July 2009 study is the first application that actively applied and assessed the five elements of the ILT 
(Sistek-Chandler, et al, 2009). Students responded to using the ILT without prompting or hesitation. Although the 
population for the initial study was small (n=12), the pilot proved to be the resounding utility of the ILT and the 
need to include this application and analysis in future EDT 610 classes. Since the initial pilot, an additional 40 
students have been asked to participate in the study of the ILT to analyze a sample blog and reflect upon their 
understanding of the ILT stages. 

Application of the Interactive Learning Taxonomy in Future Studies 

In applying the interactive learning process, we believe this approach has the potential to produce a high- 
quality online learning environment that actively and purposefully engages learners (Baldwin & Sabry, 2003). 
According to Steinaker & Leavitt (2008), the strength of interactive learning is that it encapsulates the overall 
process of learning and helps to provide a concrete representation of knowledge for the student. Implementation of 
the ILT theory becomes the springboard to new areas of learning and assists the learner to move toward the 
replication of the process in a new context. The taxonomy of interactive learning can be utilized for online teaching 
while the designer applies each step of the ILT to a planned and purposeful pathway for learning. 

Online learning environments (OLE) and online content is typically designed for one-way dissemination of 
information (Hughes, Terveen, Ernst, & Ooms, 2009). Hughes et al posit that the OLE delivers the instructor’s 
content and structure, without the consideration of students’ needs, perspective, or interests. We refer to this as flat, 
passive content with a minimum level of interactivity. Our research extends beyond the issue of learning style and 
emphasizes the need to apply 5 guiding principles from the ILT to engage the learner with interactive content; 
content that promotes interactivity between teacher and students, between students and resources, and among 
students. The Web interface is a bridge between instruction and learning (Cassarino, 2003). It is clear that while 
presentation of content is clearly one important aspect of any learning encounter, without effective interactivity 
manifested through communication, involvement, control and adaptation, the effectiveness of online and flexible 
learning will be minimized (Sims, 2003). 

The next step in the investigation of the use of this taxonomy is to solicit suggestions to refine the ILT 
categories. As we move forward with new classes of EDT610, we plan to have the students use this taxonomy (ILT) 
to further analyze not only the work of a peer which serves as a model online lesson/blog but also to apply this same 
analysis as a reflection on their own instructional, online content (blog). 
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As computer use and the internet have become more mainstream so has the acceptance of utilizing the 
internet as an alternative platform to perform many tasks that have traditionally been in physical environments. 
Considering how the world has embraced the internet, it is no surprise educational institutions have found the web to 
be a good home for the next evolution in distance learning. While technology has been the driving force behind most 
of the evolutions in distance education, it is also the force behind the generally recommended good practices of 
online pedagogy (Moore& Kearsley, 2005; Belderrain, 2006). It is now considered status quo for an online learning 
course to have a computer-mediated communication medium in which students collaborate, interact, and engage in 
discussions. Most commonly, this mediated communication takes place asynchronously inside the learning 
management systems by means of an electronic bulletin board, also commonly known as discussion boards (Hew & 
Chung, 2008). Through this mediated communication students are afforded the opportunity to establish connections 
with their peers through communicative exchanges. These dialogues may be socially driven or they may be driven 
by the content of the instruction. 

As discussion boards become the de facto standard medium for peer communication in online learning 
(Dennen, 2005), the next generations of web tools has come of age. Web 2.0 is a term used to describe a new variety 
of online tools that are geared toward enabling and promoting user participation in the creation of the web content 
(O’Reilly, 2005). The key to Web 2.0 is a sense of participatory media and increased collaboration and online 
community. With Web 2.0 and good practice for online learning sharing similar theoretical frameworks is seems as 
though this next generation of applications would be a good match with online education. Would a Web 2.0 utility, 
such as an online blog, provide the same learning opportunities as electronic discussion boards? According to 
Christenson, Anakwe, and Kessler (2001) providing a computer-mediated online learning environment is not 
sufficient for a technology to be seen as inherently viable. A technology must be considered by students’ to be 
useful, in general, and specifically useful to online learning efforts (Christenson, Anakwe, & Kessler, 2001). The 
potential usefulness of a specific type of Web 2.0 application, blogs, can be found in the structure. Blogs are simply 
an evolution of the traditional web page; compromised of individual posts, rather than presenting information as web 
pages. A blog post may only be a single commentary no more than a couple sentences or a paragraph or it may be 
much more lengthy and detailed, some may even incorporate audio and video segments. The opportunities for each 
student to post substantive comments to other students’ blog entries add an additional tier or interactivity and social 
engagement (Glogoff, 2005). In online learning communities enhanced opportunity to promote collaboration and 
interaction is highly beneficial to learners. 

Although asynchronous discussion boards have become standard parts of online learning courses, online 
learners are becoming more accustomed to Web 2.0 technologies. Would student interaction and collaboration be 
different if online students used blogs instead of the standard message boards? Does the format of the interaction 
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make a difference in the level of learning that takes place for students? Jung, Choi, Lim and Leem (2002) found that 
increased collaboration activities among students did not lead to an increase in learning among the students. 
Similarly, Godwin, Thorpe and Richardson (2008) report adoption of interactive computer-mediated learning within 
online learning environments may not be enough to lead to positive learning outcomes. 

The very process of writing encourages reflection, which helps promote higher level learning so computer- 
mediated collaboration should enhance student learning (Hew & Chung, 2008). Results from other studies found 
that students have a preference for more collaborative learning environments (Christenson, et al. 2001). While some 
research suggests that learners can benefit from simply reading the substantive postings of other students and taking 
an active role in collaborating may not be necessary for all students in distance learning courses (Gulati, 2008). 
Results reported by Hara, Bonk and Angeli (2000) found mediated communication platforms in online learning 
courses lead to increased substantive collaboration among students as well as higher cognitive processing of 
instructional materials, as based on posting analysis. Dennen (2005) similarly found online courses with mediated 
communication forums where students had a higher level of participation and less socialization focus also had a 
higher quality of cognitive learning expressed via message postings. While Fernandez (2007), concluded that social 
presence in the online community can aid in collaboration and learning. 

This paper examines the results of an experimental research study aimed at answering the following 
question: What effect does the format of computer-mediated communication have on students’ learning and 
substantive interaction in online asynchronous distance learning courses? The dependent variables of the study 
included student interaction, as indicated through qualitative analysis of the message posting by utilizing the 
definition of meaningful interaction posed by Seo (2007). The second dependent variable was learning, as indicated 
through analysis of the message postings qualitatively assessed by means of according to Krathwahl’s (2002) 
revised version of Bloom’s (1977) taxonomic framework and through posttest measures of learning objectives. 

Methods 

Participants 

The participants for this study are twenty graduate-level education students from an educational technology 
course at a large public university in the southwestern region of the United States. The course focused on the 
integration of technology in education. The ages of the participants range from approximately 23 to 52 years old. 
Sixty percent of the participants were female and forty percent of the participants were male. All students had 
previously completed at least one online course. All students had the minimum baseline level of technical 
competency required for participation in an online distance-learning course. Examples of these technical 
competencies include: accessing and receiving facilitation through an online learning management system, 
communicating with other students and their instructors using emails and discussion boards, and participating in 
distance learning assessments, such as quizzes and exams, in the online modality. 

Materials 

Instructional module 

A module on designing online instruction was presented in the Blackboard LMS. Module objectives 
included 1) Discuss the types of Websites that can be used in educational settings, 2) Explain what planning is 
necessary prior to establishing Web-based instruction, 3) Understand how HTML editing software is used to create 
Web pages, 4) List strategies for organizing the documents of a Website, 5) Explain why regular maintenance is 
vital to a quality Website. Several reading assignments, a learning activity, and a discussion prompt were provided 
as a means for students to engage in the module’s conceptual information. 

Discussion Board 

All students received the same prompt and repeated instructions about the number of required posts; one 
original posting and two responses to other postings were the typical expectations for computer-mediated discussion 
in the course. The posting expectations were to write one or two paragraphs about the educational tool and how the 
tool could be most effective in teaching/training and learning. Students were directed to suggest a technology tool 
related to the chapter reading for the week, which would be effective in teaching/training and learning. They were 
also prompted to provide a link to the tool's website and a link to at least one demonstration or tutorial available 
online for the selected tool. 
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A discussion board rubric designed to assess cognitive performance was adapted from a similar rubric 
created by Christopher, Thomas, and Tallent-Runnels (2004). This rubric assigned point values to the cognitive 
performance demonstrated in the students’ postings as delineated by categorization into in low, mid, and higher 
order thinking skills as displayed in Table 1. The basis of the rubric was formed using Krathwahl’s (2002) revised 
version of Bloom’s taxonomy cognitive process dimensions 

Table 1 



Cognitive Performance Rubric 



Score 


Dimension 


Explanation 


Related Thought Processes 


1 


Remember 

Understand 


Learner must be able to recall 
information such as dates, events, 
places, ideas, definitions, formulas, 
theories, etc. 

The learners must be able to grasp 
the meaning of the information, 
express it in their own words, 
and/or cite examples. 


Recognizing, Recalling, Identifying, Retrieving, 
Clarifying, Paraphrasing, Representing, Translating, 
Exemplifying, Illustrating, Instantiating, Classifying, 
Categorizing, Subsuming, Summarizing, Abstracting, 
Generalizing, Inferring, Concluding, Extrapolating, 
Interpolating, Predicting, Comparing, Contrasting, 
Mapping, Matching, Explaining 


2 


Apply 

Analyze 


The learner must be able to use or 
apply knowledge or skills to new 
situations. The learner must be able 
to use information and knowledge 
to solve a problem, answer a 
question, or perform another task. 
The learner must be able to break 
down knowledge into parts, and 
show and explain the relationships 
among the parts. 


Executing, Carrying out, Implementing, Using, 
Differentiating, Discriminating, Distinguishing, 
Focusing, Selecting, Organizing, Finding coherence, 
Integrating, Interpreting, Outlining, Parsing, 
Structuring, Attributing, Deconstructing 


3 


Evaluate 

Create 


The learner must be able to judge 
or assess the value of material and 
methods for a given purpose. 

The learner must be able to pull 
together parts of knowledge to 
form a new whole and build 
relationships for new situations. 


Checking, Coordinating, Detecting, Monitoring, 
Testing, Critiquing, Judging, Generating, 
Hypothesizing, Planning, Designing, Producing, 
Constructing, Constructing models 



The definition of meaningful interaction in CMC for educational exchanges introduced by Seo (2007) was 
utilized to classify interaction among participants in the two types of CMC. Postings that (a) pertained to a 
discussion topic, (b) responded to a question or an idea expressed in a previous statement or invites a comment, and 
(c) enriched the conversation by adding substance to the discussion, were coded as fully interactive. A rubric was 
created using this scale to code the students’ postings and is illustrated in illustrated in Table 2. 



Table 2 

Substantive Interaction Rubric 



Score 


Criteria 


1 


Pertained to a discussion topic 


2 


Responded to a question or an idea expressed in a 
previous statement or invites a comment 


3 


Enriched the conversation by adding substance to the 
discussion 
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Assessment 

Demonstration of student comprehension of content presented in the learning module was assessed via a 
ten-question multiple-choice quiz delivered through the course LMS. The questions for the quiz aligned with the 
learning objectives for the module, the module learning activities and the weekly assigned readings. All questions in 
the quiz related to the learning module topic of designing online instruction. 

Students were also presented with an attitudinal survey to gauge technology preference, receptivity to 
technology, confidence with technology, and frequency of technology use. The attitudinal survey was adapted from 
a previously validated instrument created by Christensen, Anakwe, & Kessler (2001). The adaptations in the survey 
represent evolutions in technology and distance learning advancements. 

Procedures 

This was a two-part, quasi-experimental study. The study was conducted using participants from a graduate 
level, education course. All students had previously participated in a learning module on Web 2.0 technologies and 
had at least one previous experience interaction with blog technology. All students in the course were expected to 
participate in group “discussion activities” via the discussion board in the Blackboard LMS each week of the 
semester. The instructions for participation in the discussion activities were presented in the syllabus and reiterated 
by the instructor via postings in the Blackboard LMS. 

The first part of the study was conducted as a formative evaluation of materials. In this first part of the 
study, fifteen students from a graduate level education course were randomly assigned to one of the two treatments 
groups, discussion board or blog. Students completed the discussion activities, the module quiz and the attitudinal 
survey. Adjustments to the study instalments and methodologies were made after observing participants and 
reviewing feedback from the formative evaluation phase. 

In the second phase of the study, twenty students from a graduate level education course were randomly 
assigned to one of the treatment groups, discussion board or blog. All students had previous experience with online 
learning, discussion boards and blogs. The study took place during the seventh week of the semester. All students in 
the course were introduced to Web 2.0 technologies as a regular part of the course instruction during the fifth week 
of the course. One instructor facilitated the learning for the course in which all the participants were enrolled. That 
instructor served as the study liaison and answered any question from the students that arose including providing 
directions to find the link to the blog and the attitude survey. The instructor did not provide any feedback on the 
requirements or expectations for postings in either of the mediated formats during the tenure of the study. The 
students who were randomly selected to participate in the blog groups received and email invitation to the blog 
location sent via the Blackboard LMS. The link to the blog was also placed in the learning module materials along 
with the link for the discussion board for the week, the online quiz and the attitudinal survey. 

Measures 

This research study uses both quantitative and qualitative approaches to describe and analyze patterns of 
participation, interaction, and meaning construction in CMC postings. Following measures validated by Christensen, 
Anakwe and Kessler (2001) the distance learning questionnaire was used to assess student preferences toward 
distance learning, preference for various types of CMC and frequency of use of CMC tools. The questionnaire 
includes a collection of demographic data and a survey about the receptivity toward and preference for the method 
of CMC they used during the study. The questionnaire was adapted to include current technology references as well 
as questions related to the learning module. 

The content was analyzed to consider the level of cognitive performance and substantive interaction 
displayed within CMC postings. The rubrics developed follow the work initially introduced by Henri (1990) who 
utilized levels of Blooms taxonomy to categorize deepness of cognition. The rubric created in this study was adapted 
from Christopher, Thomas and Tallent-Runnels (2004) using a three level system to assess cognitive performance 
via the cognitive process domains as indicated in Krathwohl’s (2002) revision of Bloom's Taxonomy. 

The definition of meaningful interaction in CMC for educational exchanges introduced by Seo (2007) was 
utilized to classify substantive interaction among participants in the two types of CMC. Messages that (a) pertained 
to a discussion topic, (b) responded to a question or an idea expressed in a previous statement or invites a comment, 
and (c) enriched the conversation by adding substance to the discussion, was coded as fully interactive. The lead 
researcher twice scored each of the postings according to the substantive interaction rubric. 

The posttest measured students’ understanding of the learning module content. The quiz questions were adopted 
from the instructor textbook companion website provided by the textbook publisher and aligned with the module 
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objectives. The ten questions were all in multiple-choice format and students all received the same set of questions 
on the quiz. 

Research Design 

This study is a one-by-two posttest only design. The independent variable was computer-mediated 
communication format (blog or discussion board). The dependent variables were student interaction in postings and 
learning, as measured by cognitive performance in postings and quiz performance, as well as student attitudes. 

Results 

This section summarizes the differences between treatment groups related to student cognitive performance and 
substantive interaction in discussion board postings and performance on posttest measures. 

Cognitive performance 

The means and standard deviations for cognitive performance by mediated format (discussion board or 
blog) are presented in Table 3. The mean score for students in the blog prompt treatment was 1.91 (SD =.944) 
compared to the mean score of 1.00 (SD =.866) for students in the discussion board treatment group. The overall 
mean score for student cognitive performance was 1.50 (SD=1.00). 

Table 3 



Means and Standard Deviations for Cognitive Performance by 
Mediated Format Condition 





Blog 


Discussion Board 


Total 


Mean 


1.91 


.944 


1.50 


Standard Deviation 


1.00 


.866 


1.00 


N 


11 


9 


20 



Note. The maximum possible score for the cognitive performance was 3. 



An ANOVA was conducted to assess the effect of computer-mediated format on the dependent variable, 
learning, as demonstrated through cognitive performance in discussion postings. The results are presented in Table 
4. The main effect was found to be statistically significant, F ( 1, 18) = 4.939, p = .039, partial r\2 = .215. These 
results indicate students in the blog group received significantly higher scores for cognitive performance when their 
postings were assessed as a measure of learning. 

Table 4 



ANOVA Summary > Table Cognitive Performance by Treatment 



Source 


df 


F 


Partial x\2 


P 


Contrast 


1 


4.939 


.215 


.039 


Error 


18 









Note. The mean difference is significant at the .05 level. 



Posttest performance 

The means and standard deviations for posttest performance by mediated format (discussion board or blog) 
are presented in Table 5. The mean score for students in the blog prompt treatment was 2.82 (SD =3.027) compared 
to the mean score of 2.44 (SD =3.005) for students in the discussion board treatment group. The overall mean score 
for student posttest performance was 2.65 (SD =2.943). 
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Table 5 

Means and Standard Deviations for Posttest Score by Mediated 



Format Condition 




Blog 


Discussion Board 


Total 


Mean 


2.82 


2.44 


2.65 


Standard Deviation 


3.027 


3.005 


2.943 


n 


11 


9 


20 



An ANOVA was conducted to assess the effect of computer-mediated format on the dependent variable, 
learning, as demonstrated through posttest score. The main effect was not found to be statistically significant, F ( 1, 
18) = 0.076, p = .786, partial r|2 = .004. 

Substantive interaction 

The means and standard deviations for substantive interaction by mediated format (discussion board or 
blog) are presented in Table 6. The mean score for students in the blog prompt treatment was 0.91 (SD =1.044) 
compared to the mean score of 1.67 (SD =1.323) for students in the discussion board treatment group. The overall 
mean score for student substantive interaction performance was 1.25 (SD =1.209). 



Table 6 

Means and Standard Deviations for Substantive Interaction by 
Mediated Format Condition 





Blog 


Discussion Board 


Total 


Mean 


.91 


1.67 


1.25 


Standard Deviation 


1.044 


1.323 


1.209 


n 


11 


9 


20 



Note. The maximum possible score for the cognitive performance was 3. 

An ANOVA was conducted to assess the effect of computer-mediated communication format on the 
dependent variable substantive interaction. The main effect was not found to be statistically significant, F (1, 18) = 
2.053, p = .169, partial r|2 = .102. 

Survey Results 

Students were given an attitude survey that contained forty-seven Likert scale style questions and three 
open-ended questions. The survey was grouped into categories including perceived technology usefulness, 
technology usage and technology preference. Technology usefulness and technology preferences were scored on a 
one to five scale and technology usage was scored on a one to six scale. The survey also included demographic 
criteria. The means and standard deviations by mediated format (discussion board or blog) are presented in Table 7. 

The items that received the most positive responses referenced frequency of use of the World Wide Web 
and email. Students had the most positive attitude on items number thirty-six and thirty-seven, which inquired about 
frequency of technology use for the World Wide Web (M =5.80, SD =0.632) and frequency of e-mail usage (M 
=5.80, SD =0.422 indicating that all repondants used each of the technologies everyday. The item that received the 
next most positive response was item number twenty-three, which asked students how many times every day they 
used the world wide web (M =5.30, SD =1.059) indicating that all respondents used the web several times everyday. 

The three items that received the most negative responses each referenced frequency of technology usage. 
Students had the most negative attitude on item number twenty-nine pertaining the amount of time spent on a daily 
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basis using chat rooms (M =1.00, SD = 0.00) indicating that all students almost never use chat technology. The item 
that received the second most negative response was item number thirty pertaining the amount of time spent on a 
daily basis using blogs (M =1.40, SD =.699) indicating that students on average use blogs for less than thirty 
minutes each day. 



Table 7 

Means and Standard Deviations for Survey Items 



Total 



Item 






Mean 


SD 


Mean 


SD 


1 


I prefer to use blogs. 


Blog 


2.50 


.837 


2.50 


.707 






Discussion Board 


2.50 


.577 






2 


I prefer to use discussion boards. 


Blog 


3.67 


1.033 


3.70 


8.23 






Discussion Board 


3.75 


.500 






3 


Using a blog in a class helps me learn 


Blog 


3.33 


.816 


3.30 


.675 






Discussion Board 


3.25 


.500 






4 


Using a discussion board in a class helps me learn 


Blog 


4.33 


.816 


4.20 


.632 






Discussion Board 


4.00 


.000 






5 


This module was valuable 


Blog 


4.33 


.516 


4.20 


.422 






Discussion Board 


4.00 


.000 






6 


I liked this module 


Blog 


4.17 


.408 


4.00 


.471 






Discussion Board 


3.75 


.500 






7 


I would consider distance learning for a core 


Blog 


4.33 


.816 


4.10 


.994 




course 


Discussion Board 


3.75 


1.258 






8 


I would consider distance learning for an elective 


Blog 


4.17 


.753 


4.30 


.675 




course 


Discussion Board 


4.50 


.577 






9 


How useful do you feel the world wide web would 


Blog 


2.00 


.632 


2.30 


.823 




be for distance learning 


Discussion Board 


2.75 


.957 






10 


How useful do you feel e-mail would be for 


Blog 


3.50 


1.049 


3.70 


.949 




distance learning 


Discussion Board 


4.00 


.816 






11 


How useful do you feel video conferencing would 


Blog 


3.83 


.753 


3.80 


.632 




be for distance learning 


Discussion Board 


3.75 


.500 






12 


How useful do you feel chat rooms would be for 


Blog 


3.83 


.753 


3.60 


.843 




distance learning 


Discussion Board 


3.25 


.957 






13 


How useful do you feel audio tape would be for 


Blog 


2.33 


1.366 


2.50 


1.179 




distance learning 


Discussion Board 


2.75 


.957 






14 


How useful do you feel video tape would be for 


Blog 


2.67 


1.211 


3.10 


1.101 




distance learning 


Discussion Board 


3.75 


.500 






15 


How useful do you feel US mail would be for 


Blog 


1.50 


.837 


2.10 


1.197 




distance learning 


Discussion Board 


3.00 


1.155 






16 


How useful do you feel discussion boards would 


Blog 


4.17 


.753 


4.30 


.675 




be for distance learning 


Discussion Board 


4.50 


.577 
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17 



.843 



18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 



How useful do you feel blogs would be for 
distance learning 

How useful do you feel wikis would be for 
distance learning 

How useful do you feel podcasts would be for 
distance learning 

How useful do you feel file sharing communities 
(such as YouTube) would be for distance learning 

How useful do you feel Virtual Worlds (such as 
Second Life) would be for distance learning 

How useful do you feel Social Networks (such as 
Facebook & MySpace) would be for distance 
learning 

How much time you spend on a daily basis using 
the world wide web 

How much time you spend on a daily basis using 
e-mail 

How much time you spend on a daily basis 
transferring files with a computer 

How much time you spend on a daily basis using 
video conferencing 

How much time you spend on a daily basis using 
discussion boards (academic) 

How much time you spend on a daily basis using 
discussion boards (non-academic) 

How much time you spend on a daily basis using 
chat rooms 

How much time you spend on a daily basis using 
blogs 

How much time you spend on a daily basis using 
wikis 

How much time you spend on a daily basis using 
podcasts 

How much time you spend on a daily basis using 
file sharing communities (such as YouTube) 

How much time you spend on a daily basis using 
virtual worlds (such as Second Life) 

How much time you spend on a daily basis using 
social networks (such as Facebook & MySpace) 

How frequently do you use the world wide web 



Blog 


3.83 


.753 


3.60 


Discussion Board 


3.25 


.957 




Blog 


4.00 


.632 


4.00 


Discussion Board 


4.00 


.816 




Blog 


3.67 


1.033 


3.60 


Discussion Board 


3.50 


1.291 




Blog 


3.67 


1.033 


3.60 


Discussion Board 


3.50 


1.000 




Blog 


2.67 


1.506 


2.70 


Discussion Board 


2.75 


.500 




Blog 


2.83 


1.602 


3.00 


Discussion Board 


3.25 


1.258 




Blog 


5.17 


1.329 


5.30 


Discussion Board 


5.50 


.577 




Blog 


4.33 


1.633 


4.40 


Discussion Board 


4.50 


1.291 




Blog 


3.00 


.894 


3.30 


Discussion Board 


3.75 


1.258 




Blog 


4.17 


3.125 


2.90 


Discussion Board 


1.00 


.000 




Blog 


3.00 


1.673 


3.10 


Discussion Board 


3.25 


.957 




Blog 


2.00 


1.673 


2.10 


Discussion Board 


2.25 


1.500 




Blog 


1.00 


.000 


1.00 


Discussion Board 


1.00 


.000 




Blog 


1.50 


.837 


1.40 


Discussion Board 


1.25 


.500 




Blog 


1.50 


.548 


2.40 


Discussion Board 


3.75 


2.363 




Blog 


2.83 


2.401 


2.10 


Discussion Board 


1.00 


.000 




Blog 


1.83 


.983 


2.50 


Discussion Board 


3.50 


2.380 




Blog 


3.33 


2.944 


3.50 


Discussion Board 


3.75 


3.202 




Blog 


2.17 


1.472 


2.30 


Discussion Board 


2.50 


2.380 




Blog 


5.67 


.816 


5.80 


Discussion Board 


6.00 


.000 





.667 

1.075 

.966 

1.160 

1.414 

1.059 

1.430 

1.059 

2.846 

1.370 

1.524 

.000 

.699 

1.838 

2.025 

1.780 

2.877 

1.767 

.632 
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37 


How frequently do you use e-mail 


Blog 


5.67 


.516 


5.80 


.422 






Discussion Board 


6.00 


.000 






38 


How frequently do you transfer files with a 


Blog 


4.33 


1.506 


4.40 


1.776 




computer 


Discussion Board 


4.50 


2.380 






39 


How frequently do you use video conferencing 


Blog 


4.50 


2.950 


3.70 


2.983 






Discussion Board 


2.50 


3.000 






40 


How frequently do you use discussion boards 


Blog 


2.67 


2.422 


2.90 


2.378 




(academic) 


Discussion Board 


3.25 


2.630 






41 


How frequently do you use discussion boards (non 


Blog 


3.83 


1.602 


4.20 


1.398 




academic) 


Discussion Board 


4.75 


.957 






42 


How frequently do you use chat rooms 


Blog 


2.17 


2.401 


2.30 


2.497 






Discussion Board 


2.50 


3.000 






41 


How frequently do you use blogs 


Blog 


3.33 


2.582 


3.20 


2.530 






Discussion Board 


3.00 


2.828 






43 


How frequently do you use wikis 


Blog 


4.17 


2.041 


4.40 


1.955 






Discussion Board 


4.75 


2.062 






44 


How frequently do you use podcasts 


Blog 


4.00 


2.757 


3.40 


2.797 






Discussion Board 


2.50 


3.000 






45 


How frequently do you use file sharing 


Blog 


3.33 


1.862 


4.10 


1.912 




communities (such as YouTube) 


Discussion Board 


5.25 


1.500 






46 


How frequently do you use virtual worlds (such as 


Blog 


3.33 


2.944 


3.40 


2.797 




Second Life) 


Discussion Board 


3.50 


3.000 






47 


How frequently do you use social networks (such 


Blog 


3.17 


2.483 


3.90 


2.183 




as Facebook & MySpace) 


Discussion Board 


5.00 


1.155 







Discussion Board 

Note. The maximum possible score for preference section was 5 with higher scores indicating stronger agreeance. 
The maximum possible score for usefulness section was 5 with higher scores indicating more perceived usefulness. 
The maximum possible score for usage and frequency was 6 with higher scores indicating more frequent usage. 

A multivariate analysis of variance (MANOVA) was conducted on student attitude levels toward 
technology preference. The effect was not found to be statistically significant, Wilk’s Lambda = .461 F (1, 8) = .146, 
p = .969, partial r\2 = .539. 

Data was collected from a set of open-ended questions in both phases of the research. One of the questions 
related to technology preference regarding blogs and discussion boards. The other open-ended questions related to 
satisfaction with the learning module. The responses from the technology preference section are displayed in Table 
8 . 



Table 8 

Open-ended responses to the prompt, “ Please explain your answers [regarding technology’ preference] in more 
detail” 

The blog was new to me, so it was a little harder to use. I have used blogs before, but don't like them. They don't 
seem as "organized" as discussion boards with post titles, response trees, etc. 

We used a blog . . . and I found it unwieldy and unorganized. With a discussion board there is a thread with a clear 
topic and it is organized and easy to follow a blog is too all over the place for me. A blog also feels more social 
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than educational to me. Perez Hilton has a blog, he doesn't have a discussion board. A discussion board by nature 
seems more educational to me. 

1 have no experience with blogs yet. I like the discussion board because 1 learn from the other students and enjoy 
reading their posts, but I think that the instructors should have to respond to each of our posts if you're going to 
assign one every week. They are a lot of work. At this stage I am more comfortable using the discussion board, 
but with a little time 1 feel sure I would be just as comfortable with using a blog. 

1 have never used blogs in an online class, but I don't think they would be all that effective. Discussion boards just 
seem so tedious. I sometimes feel like 1 just do them to get them done. 

Discussion boards lead to more interaction. They mimic a conversation more than a blog. Blogs are more like the 
long ramblings of the student who can't make their point. They can be interesting and informative but not 
interactive. 

I have noticed that discussion boards are shorter than blogs. But I justed created my onw blog and I lust love it. 

1 think both are useful tools in discussing, 1 just prefer the blog format 

I think a blog is more interactive rather than the discussion board which seems very choppy and less like a 
discussion, 

I do not really see a difference between using a blog and/or the discussion board. I will add, however, that blogs 
are more aesthetically pleasing. 

I feel that discussion boards foster more natural conversation, as opposed to blogs, which are primarily one-sided. 

Discussion boards for me are not necessarily learning opportunities as much as they are a chance to just see how 
other people think. The one exception was my BIO class . . . where we used it to answer questions for the 
upcoming test. Going through that and answering other students' questions helped ME learn the material and was 
really gratifying. :) 

I have been in learning situations where the usefulness of either of the above tools is determined by how the 
instructor sets them up. 



Discussion 

The results of this study did not indicate a significant difference in learning as measured by the posttest. 
However, the qualitative assessment of postings indicated a significant difference in learning, as measured by 
cognitive performance using a scale based on Blooms revised taxonomy (Krathwohl, 2002; Christopher, Thomas, & 
Tallent-Runnels, 2004). Results showed that participants assigned to the blog group composed posts that were rated 
at a higher cognitive level than students who created postings in discussion board format. There was no significant 
difference in the level of substantive interaction between the students in either of the computer-mediated formats. 
Although the participants assigned to the discussion board group posted more responses than the students in the blog 
group, the postings were not qualified as substantive by measures of this study (Seo, 2007). This indicates that the 
form of communication in which students were assigned to post in this study did not make a difference in how 
thoroughly they interacted with their peers in the posting assignment. 

The posttest results did not yield any significant difference between the mediated format groupings. The 
mean score for the posttest did not yield significant results when used as a measure of learning. 

Feedback gathered from the open-ended questions indicates that some students may see posting 
assignments as tedious in any format. Students who were less familiar with the blog technology appear to indicate a 
lack of comfortableness with the medium and indicated that it could be found to be less organized and more one- 
sided than blogs. However, other students mentioned the aesthetically pleasing nature of blogs and indicated a 
growing fondness for the technology. Open-ended feedback also indicated students may find regular posting 
assignments to be tedious, in general. Respondents referred to instructor interaction and feedback as a criterion in 
satisfaction with the technology used to facilitate interaction. 

Reponses on the survey, not surprisingly, show regular use of the internet and e-mail among the student 
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participants. Mean preference scores also indicated that blog use was rated as one of the lowest among types of daily 
technology usage, while academic discussion boards showed a higher reported daily usage among the subjects of the 
study. This would support the notion that discussion boards are the most typical type of technology used in distance 
learning courses to simulate classroom interaction (Dennen, 2005; Hew & Chung, 2008). 

Limitations of study 

The small sample size was one of the biggest limitations of this study limiting the power and 
generalizability of the findings. The length of the attitudinal survey may have also limited the results, as some of the 
participants did not complete all of the responses. 

The blog was external to the Blackboard LMS and it is possible that the additional effort required to access 
the blog was demotivating to some of the participants. Disproportionate exposure to the computer-mediated 
technologies used in this study may confound the performance and feedback from the participants. 

The study was conducted in an Educational technology course. This could also have an impact on the 
results, as these students may be more generally accepting of technology, especially for the purposes of learning, 
than the general student population in other educational disciplines. 

The study participants reviewed the overall satisfaction with the learning module favorably but the 
performance on the posttest was low for all the students. The assigned prompts used in this study were loosely 
associated with educational technology tools in general, rather than the specific technology addressed in the learning 
module. It is possible that performance on posttest measures may have been different if the content were more 
tightly associated with the content of the discussion prompt and the assigned readings. 

Implications 

Usage of technologies, such as blogs, for peer interaction in distance learning may lead to higher cognitive 
performance on learning objectives. Varying the design of a distance learning course beyond the typical usage of 
discussion board for regular posting assignments and student interaction may yield a gain in learning for students 
who compose more thoughtful postings in an alternate medium. 

Distance learning facilitators and designers may want to consider integrating instructor feedback and more 
structured cognitive challenges for student posting assignments. Generalized prompts that are aimed to lead toward 
regular student interaction may not be found to be academically stimulating for students in distance learning courses. 
Participants in this study indicated an interest in interacting with the facilitator as well as other students during 
weekly computer-mediated interactions. 

Future Research 

Future research to build upon the results of this study may consider the addition of instructor feedback as a 
variable. Further, future studies would gain insight by having a larger sample of participants from which to collect 
data. Many graduate level courses have a smaller enrollment size than do undergraduate level courses. Although 
graduate students do tend to be more intrinsically motivated to complete assignments, an undergraduate level course 
with high enrollment may serve as a better population from which to draw a sample. Research with students who are 
not in the education field would also provide insight into the interaction of students in other disciplines. The 
prevailing pedagogy and design of the distance learning courses in other disciplines may differ significantly from 
that of a graduate level education course. It may also be valuable to further dissect the information about students’ 
postings and interaction in the mediated formats by categorizing initial postings and response postings separately. It 
is possible the level of interaction and cognitive performance differs depending on whether a student is composing 
an initial post or responding to others. It may also be of interest to examine demographic difference between 
subjects for familiarity with and preference for specific learning technologies. It is possible that gender and age may 
bias learners toward certain types of computer mediated communication technologies, which could be useful when 
designing distance-learning coursework. Another criteria for consideration of inclusion in an attitudinal survey may 
be the concept of passive learning gained from reading the postings of other students even if the reader does not 
respond or interact with the postings. The passive cognitive gains from studying these interactions may provide 
insight into the collective learning communities that form in distance learning courses. 
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Conclusion 



Communication in distance learning courses is an important part of team building and collaboration that 
helps simulate the natural community of the traditional classroom. Discussion boards contained within the learning 
managements system that facilitate the distance learning course quite often default to built-in discussion boards for 
the purposes of peer interaction and collaboration. Other mediated communication options, such as blogs, may prove 
to provide a higher level of cognitive performance and could lead to more substantive interaction amongst 
participants. The results of this study indicate that students who used a blog for a weekly posting demonstrated 
higher levels of cognitive performance in their postings. 

Higher levels of cognitive performance are often associated with gains in learning. It is possible that 
deviating from the typical computer mediated format may stimulate students thinking about discussion prompt. It is 
also possible that once communication is out of the learning management system and instead out on the internet for 
others to see mundane assignments may be taken more seriously. Consideration should be given to designing a 
distance-learning course with the audience in minds. Technology novices and those who are not adept at learning 
new technologies may not find a new communication format engaging. They instead, may find the extraneous 
cognitive load to be detrimental toward the cognitive performance and substantive interaction that is beneficial for 
student learning in computer mediated communication assignments. 
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Abstract: The number of Facebook users has dramatically increased in several countries. 
The highest number of users are in the United States, followed by the United Kingdom, 
Turkey, and Indonesia (GreyReview, 2010). Issues involving its use have increased as the 
number of users reach the 500 million mark (Facebook Press Room Statistics, 2010). As a 
social network, Facebook provides an exciting interactive online environment. Users 
collect “friends” as a manifestation of their increasing popularity, share their lives through 
photographs, reconnect with old friends, and maintain friendships. However, users are also 
open to online attacks, diminished real-life social skills, and identity theft. This paper 
attempts to initiate a critical discussion and establish relevant research methodology for 
studying adolescent awareness of ethical and security issues in Indonesia. The authors, 
users of Facebook as well as parents and teachers of Facebook users, offer their personal 
experiences to anchor the discussion. 



Introduction 

Facebook (FB), considered the most popular social networking website today, has raised many concerns 
regarding its use by 18 - 24 year-old young adults and 13 - 17 year-old adolescents (Webster, 2008). Most youth, 
excited by its features and applications, constantly check their FB pages and have difficulty staying away for long 
periods of time (Mori, 2009). The integration of FB on smart phones, like the iPhone, has exacerbated the situation 
by giving users instant access from any location (Watkins, 2009). However, FB’s popularity among youth has drawn 
concern and criticism from parents and educators (Goodman, Maggio, & Lyman, 2008). The publishing and sharing 
of personal information on FB make malicious attacks possible, real-life socializing less common, and increase the 
risk of identity theft by strangers (Wang, 2009). 

In the United States, online behavior and security are critical issues due to a proliferation of malicious 
behavior and cyber-crime. Online insults create conflicts between adolescents, adolescents and parents, and 
adolescents and teachers. Identity theft, cyber-bullying, and sexual harassment add to the negative consequences of 
FB misuse in the U.S. With the rise in popularity of FB in Indonesia, there is great potential for similar problems 
among Indonesia’s youth. 
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The number of FB users in Indonesia is increasing dramatically, similar to what is seen in the United 
States, United Kingdom, and Turkey. Based on The Facebook.com survey from March 1, 2010 (GreyReview, 2010), 
Indonesia was ranked 4th of the highest FB users in the world. The number of FB users in Indonesia now has 
reached near 19 million. From that number, the largest group of users are young adults (18-24) at 40.55% and 
adolescents (13-17) at 27.25%. 

Research shows social networking sites, such as FB shape an adolescent’s identity by facilitating the ability 
to network and access information (Bonk, 2009; Tapscott, 2009, Watkins, 2009). Most FB users post photos and 
videos, send messages, and chat. Many adolescents post everything on FB and include feelings, thoughts, and 
personal information, although Watkins (2009) found that not all the posted information was true. 

Adolescent FB users are generally less aware of the risks of FB use (Goodman, Maggio, & Lyman, 2008). 
Teachers and parents try to protect their children and students by banning the use of FB. However, this does not 
solve the problem. According to James, et al. (2009), it woidd be more important for adolescents to have knowledge 
about the ethical uses, effects, and risks of this media. Raising adolescents’ awareness regarding ethical online 
behavior and the importance of securing personal information will help adolescents make good decisions while 
using FB. 

Although FB has improved its security system, the responsibility remains with the user to adapt and set up 
the new security system. Unfortunately, not all users are aware of FB’s improvements and personal information 
continues to be shared freely on the Internet. 

This paper examines information from Indonesian student FB users, parents and teachers in Hawaii, and 
includes a review of the literature. With a focus on ethical and security issues, the authors seek to use this 
discussion to study adolescent FB use in Indonesia. 



Purpose 

The purpose of this paper is to: 1) Review the literature on FB security; 2) describe reasons for focusing on 
adolescents in Indonesia; 3) develop a critical body of information on the topic; 4) plan a study of FB use among 
Indonesian adolescents. 



Methodology 

Information was collected from a pilot project on Indonesian students use of FB, teacher and parent written 
perspectives of FB use by young people in Hawaii and the United States, and a review of the literature. The 
overview describes the reasons for focusing research on adolescent FB users in Indonesia and presents data from a 
brief pilot study completed previously. A description of three categories of adolescent FB users in Indonesia 
follows. Hawaii and U.S. perspectives, in the form of personal essays, are also included. A literature review 
includes case studies, articles, newspapers, and publications. The authors met face-to-face and online through 
Google Docs to discuss ideas, focus the topic, and collaborate on the writing of this paper. 

Review of the Literature 

There are both negative and positive aspects to FB (Dwyer, Hiltz, & Passerini, 2007; Ellison, Steinfield, & 
Lampe, 2007; Boyd, 2008; Boyd & Ellison, 2008). Adolescents have fun interacting with friends and keeping in 
touch with friends or relatives who live far away. On the other hand, adolescents receive inappropriate photos or 
comments and suspicious calls and messages. There is also the potential of misuse if the user forgets to log out when 
using FB in an Internet cafe. 

FB does provide users information about their rights and responsibilities as well as terms of service 
(Walters, 2009; Shih, 2009; Vander Veer, 2008; Vander Veer, 2009). This information is put on a page that is easily 
accessed by clicking the icon link at the bottom of a FB account page. In addition, when FB launched its new 
application, an email was posted to the user’s account and information was posted in a blog online (O’Neil 2009). 
When FB launched its application for users to set up their profde security, the developers post an email and publish 
information on the Internet to let users know about it. Adolescents who ignored the message and/or did not look at 
the announcement email from FB showed a “lack of awareness.” This “lack of awareness” about the security system 
was the dominant factor creating hacked or stolen FB accounts (Softmaster, 2001). Users did not secure their 
privacy and claimed that they did not know about privacy and security issues (Debatin, Lovejoy, Horn, & Hughes, 
2009). 

Security awareness means that adolescents can implement preventative actions (i.e., avoiding a friend 
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request from strangers, using the FB security system, or deleting important personal information on their accounts). 
Corrective action can include deleting the account, changing the account password, deleting the inappropriate 
posting, or making a complaint (Facebook Principles, 2007). It is important that adolescents learn and understand 
the security system and all security system updates to minimize the risks to their personal account and information. 
Brown (2005) stated that peers influenced the way adolescents think and learn about social [networking] 
relationships. It would be interesting to examine if adolescents would learn more about FB risks and etiquette if it 
came from their peers’ experiences. 

What role do parents, relatives and teachers play? Collin & Laursen (2005) tout the importance of parents 
in developing adolescent awareness. The Board of Children, Youth, and Families (2004) also reports the importance 
of teachers and school engagement. Parents, teachers and other adults have the responsibility to help adolescents use 
FB and other Internet sites safely. Adults, however, are facing difficulties with the new technology (Watkins, 2009; 
Bonk, 2009, Tapscott, 2009). A possible solution is for adults and adolescents to openly discuss Internet security 
issues together. 



Overview 

Indonesia is the world’s largest archipelago country. It covers an area of 1,919,440 square kilometers and 
contains a population of about 234 million people (Population Projections, 2010). Of the 234 million people, about 
42 million are adolescents and young adults (17.95%). Recently, 19 million people in Indonesia have started using 
FB, which is approximately 8% of the total population. Of these users, the percentage of young adult and adolescent 
FB users in Indonesia is 67.8% (GreyReview, 2010) and approximately 13 million or 5.5% of the total population. 
Based on these statistics and the first author's experience with the geography and culture of Indonesia, there are 3 
main reasons to research adolescent use of FB in Indonesia. 

Indonesian adolescent identity is shaped by ethnic and religious factors. Conformity to ethnic traditions 
and religious rules are highly valued by most Indonesians which are reinforced by families and society. Indonesian 
adolescents are raised to conform to ethnic traditions and religious rules. Since ethnic tradition and religious rules 
are the guidelines for adolescents, it will affect what they will do on FB or their actions will conflict with core 
beliefs. 

Indonesian adolescent identity is also shaped by family. Most adolescents live with family. Parents are 
more likely to have their children live with relatives or older siblings than in a dorm. In addition, family gatherings 
are one of the most important events in one’s life. Therefore, it is not unusual to include family members in FB 
accounts. These family members may shelter their adolescents from harmful behaviors in FB. 

In this study, adolescents were placed into three categories based upon access to digital technology and FB. 
The first category represents adolescents who have no access to digital technology because they are either in poverty 
or live in a remote area. This category will be excluded from this analysis because those in remote areas are most 
likely unfamiliar with FB. The second category represents adolescents with limited access to digital technology. 

This group consists of adolescents who might not have a computer and Internet access at home, but can access the 
Internet through Internet cafes. The third category represents adolescents who can easily access the Internet through 
a personal computer/laptop or a personal mobile-device (iPhone, Smartphone, etc.). While both categories share 
basic knowledge, experience, and Internet literacy, they vary in understanding and application; affecting their 
awareness of security and the ethical applications of FB use. 

The following figure shows personal perspectives about adolescent FB use (see Appendix for full essays) 
with an analysis of the resulting information that follows. 



Themes 


A 


B 
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D 


E 
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X 


X 
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Maintain Social Ties 
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X 
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X 


X 


Distraction 


X 




X 


X 




Games 




X 








Decrease in F2F Social 
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X 






Addiction 


X 




X 






School/Work Policies 


X 






X 





Figure 1. Common themes from the essays. 
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There were 7 themes that emerged, with 5 of the themes repeated in more than one essay. The theme of 
“Blurring Relationships” developed as FB is used in multiple environments - personal, work, and home. In her 
essay, B discussed the potential blurring between friends and family, as well as friend and teacher. She wrote, “I 
wondered if her friend shared that page with her mom and who else would see it that was not a “friend.” Later I 
learned that my daughter’s high school teachers were “friending” their current and former students” (para. 2). E also 
echoed the blurring between friend and family as he shared that his son is not his friend on FB (para. 2). He also 
argued that the term “friend” is too general in FB and suggested that “acquaintance” should be a “friend” alternative 
(para. 1). Similarly A wrote about the blurring between friend and work by sharing an example about a friend’s 
experience (para. 3). Through these examples, the theme of “Blurring Relationships” occurs in multiple ways. 

Two of the themes, “Distraction” and “Addiction” are connected to the frequency in which users check 
their FB page. C wrote, “At my school, Facebook is unblocked and students can, and constantly do, access 
Facebook at all times of the day” (para. 2). D’s experiences with her daughter are also similar. She shared, “Every 
waking moment my child would find some way of getting on to the site, chat with classmates who were sometimes 
sitting across, and Facebook had become the evil of every dinner conversation...” (para. 2). A expanded on this, 
sharing how FB can be a “Distraction” and an “Addiction” to working adults as well. 

Related to the frequency at which users check FB, are the policies that schools and work environment 
impose. Evidence of the “School/Work Policies” in D’s essay is shown in, “There was no monitoring system on 
student use and it became an issue of treating students like adults and having them decipher how to manage their 
time better” (para. 3). A wrote, “At work, while some networks blocked Facebook to keep their workers on track, 
there are some that don’t and I know of many who constantly check” (para. 3). Without clearly establish policies 
regarding social networking, “The temptation is there and [users are] expected to have the discipline enough to just 
say no” (D, para. 3). 

Despite the concerns that several of the authors shared, a theme that nearly all of their essays shared was 
“Maintain Social Ties (Personal and Family). This theme is actually a positive feature of FB as many users share 
information with friends to stay up to date on their relationships. D wrote, “it has kept us updated with birthdays, 
family photos, and gossip” (para. 3). B stated, “It was a way for people to keep in touch and find out what’s going 
on in the lives of friends and family” (para. 2). Although there are concerns about the safety of FB, it can 
be beneficial. FB can be a wonderful tool when users exercise restraint regarding the amount and type of 
information posted. 



Conclusion: Future Research 

In a previously completed pilot study, adolescents were placed into three categories based upon access to 
digital technology and FB. The first category represents adolescents who have no access to digital technology 
because they are either in poverty or live in a remote area. This category will be excluded from this analysis because 
they probably do not know about FB. The second category represents adolescents with limited access to digital 
technology. This group consists of adolescents who might not have a computer and Internet access at home, but can 
access the Internet through Internet cafes. The third category represents adolescents who can easily access the 
Internet through a personal computer/laptop or a personal mobile-device (iPhone, Smartphone, etc.). While both 
categories share basic knowledge, experience, and Internet literacy, they vary in understanding and application; 
affecting their awareness of security and the ethical applications of FB use. Future research would study categories 
two and three. 

Four data collection techniques can be implemented in a study of adolescent FB use. These four techniques 
include an online survey, online interview and text-chat, FB account analyzing, and participatory observation and an 
in-depth interview, which can be used to collect and analyze data. 

An online Survey is the easiest technique for collecting data with the use of survey applications like Survey 
Monkey, Lime Surveys, and Google Forms. The survey will be implemented by sending the link to adolescents’ FB 
or Gmail accounts. Potential respondents will be identified through their school affiliation. To expand the number of 
respondents, teachers and-others who-reside in Indonesia will be asked to also send the survey link. It is projected 
that a minimum of 3 months is needed to get the data representing a valid sample. It is hoped that both the data from 
adolescents who have higher access to FB and adolescents who have less access to FB can be gathered. The biggest 
challenge of the online survey will be to verify that the respondent meets the required criteria for the study, 
primarily since respondents will be anonymous. Other challenges are designing a questionnaire with the right 
amount of questions and translating it into Indonesian. 
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Interview and text-chat are two additional techniques. Skype, IM Conference Call, and Yahoo Conference 
Call can be used to facilitate a small number of follow up interviews. If the adolescents feel uncomfortable with 
talking, data can also be gathered through text-chat with FB, IM, and Yahoo chat room. More time will be needed to 
collect data with these techniques if a large number of respondents are involved. Data can also be collected from a 
small number of respondents and research can be focused on case studies. The biggest challenge in implementing 
these techniques is the language difference between most of researchers and the respondents. Other challenges for 
implementing these techniques are the process of organizing and presenting a large amount of qualitative data and 
the possibility of respondents deciding not to continue to participate in research. In case studies, the results cannot 
be generalized. 

Analyzing Indonesian adolescents FB accounts is another potential way to collect data. After gaining user 
permission, research can be started by analyzing the FB accounts accessed through the researcher’s account. To 
obtain a different access authorization, the researcher can create a new account as a non-friend. Researchers can 
collect information listed by adolescents to see what kinds of information are readily available for viewing by a 
friend and a non-friend. Comparing the different information relative to the level of friendship can give a birds-eye 
view of the number of adolescents that have implemented the security system to their accounts. 

Participatory observation and in-depth interview are the best way to collect data. Through participatory 
observation, researchers can carefully observe and record the activity of Indonesian adolescents when using FB. In- 
depth interviews will obtain more information needed for analysis. Respondents’ activities can also be recorded in a 
video. The greatest obstacle would be the long distance and the high cost needed for research. To collect the data, 
researchers need to stay in Indonesia and become involved in the adolescent’s daily life. 

A large-scale research study would add to the literature available on the topic and benefit adolescents, 
parents, and teachers by dealing with the implications of FB as a social networking site. The data would answer 
questions regarding adolescent general knowledge and responses on FB, including adolescent online activities, and 
present examples or non-examples of risky online behavior and development of social skills and ethical online 
behavior. 



Appendix 

A Student’s View of Facebook (A): Facebook began making waves while I was completing my 
undergraduate degree. I had a friend from another university who’s school was just invited, and she was so excited 
to show me. I was impressed, and couldn’t wait until my school was added. Once it hit my school, it was huge. 
Profde pages were scrutinized, photos added, friends requests sent, and more. While my profile was always 
somewhat bare, looking at everything was just so much fun. One of my roommates at the time, was still holding 
out, and with her permission I created her account. Without one, people constantly asked her to join so that she 
could be added. It was almost as if she “had” to join. 

Post graduation, things began to change. I started to go through the photos that I had been tagged in, and 
untagging myself from anything that coidd be potentially harmful. Facebook had become open to so many schools, 
that there was now potential of anyone discovering your profde, including your future boss. Some were so 
concerned that they completely deactivated their account. 

At work, while some networks blocked Facebook to keep their workers on track, there are some that don’t 
and I know of many who constantly check. This is seen with the speed at which people update their status and 
comment on one another. When Facebook completed incorporating all schools into their network, that is when the 
mixing of personal life and public life truly happened. Without the requirement of a .edu email address, anyone 
could join Facebook, and anyone could friend you and read your profde. While working, a client that a friend of 
mine was working with, friended her on Facebook. Schools were also impacted by this as well. Schools had to 
regulate teacher’s use of Facebook since student could find their profiles. An entire school district’s restrictions 
were so tough that no teacher was allowed to maintain a profde. At the college level, becoming “friends” with a 
professor is also questionable. 

Aside from concerns about mixing one’s personal and professional/academic life, is the concerns of mixing 
personal life with family life. For example, aside from having people of my generation online in Facebook, I also 
saw individuals older than me. Since I consider Facebook to be highly social, I don’t necessarily want the older 
generation of my family being privy to everything that I post or was posted about. 

Lastly, I tend to be a more private person, and don’t want the whole world to see my life moment by 

moment. 
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A Parent and Elementary School Teacher’s View of Facebook (B): I knew almost nothing about 
Facebook the day my daughter said that her college roommate was setting up a page for her. An online social 
network sounded pretty cool. 

It was a way for people to keep in touch and find out what’s going on in the lives of friends and family. It 
sounded good to me. She shared her first semester and her new friends via Facebook. I loved it! As time passed, 
there were lots of photos of performances, her study abroad program, and hiking trips. The photos were tagged with 
the names of people I came to recognize. Her friends added and tagged photos that she didn’t have. Then one day 
Facebook took on a different meaning. My daughter was looking at a friend’s page. As I looked on, I saw lots of 
pictures of parties and drinking. Hmm . . . college life? I wondered if her friend shared that page with her mom and 
who else would see it that was not a “friend.” Later I learned that my daughter’s high school teachers were 
“friending” their current and former students, and a college professor sent her a “friend” request. It seemed a bit 
awkward to me. 

Later as a teacher, 1 listened quietly as a seven-year-old student talked about Facebook’s Farmville. He 
asked if I was on Facebook and explained that he wanted to “friend” me. I smiled politely and the request was 
forgotten. Was he allowed to be on Facebook? Then I read about a class action suit fded in California against 
Facebook by parents who were angry about the likenesses of their children being used in advertising. I didn’t see a 
Facebook response about the litigation. My fears about Facebook concern minors and their ability to make good 
choices. Who are they “friending” and accepting as” friends?” Who’s seeing their pages? Where does corporate 
responsibility begin and end? What do you say to a college student who places questionable photographs of 
him/herself on a Facebook page? Scientists say that much of brain development in the area of decision-making 
happens in the mid-twenties. Mistakes on Facebook can be far-reaching. The world is large and information travels 
quickly - for good or bad. 

A Secondary School Teacher’s View of Facebook (C): As a middle and high school teacher, I think 
Facebook is in many ways hurting our students. Facebook and similar sites are called social networking sites; 
however, I strongly believe this is a skewed interpretation of what social networking really is. While it is true that 
student’s can network with their friends by “friending” them on the different websites, I sometimes wonder if the 
digital communication between friends is as effective as face to face (f2f) communication. 

At my school, Facebook is unblocked and students can, and constantly do, access Facebook at all times of 
the day. I literally see students on Facebook when I am walking to my classroom in the morning, when I’m eating 
lunch, and when I leave school at the end of the day. I’ve even seen times when a group of friends are sitting at a 
table with their laptops logged into Facebook and instant messaging their friends sitting at the same table! We 
recently had the Internet go down because of a power outage at school and it was hinny to watch the students being 
forced to make conversation with their friends. It was actually awkward for some of those students! While I am an 
Educational Technology major, and I believe in social networking, virtual reality, and even online and virtual 
learning, I also think that face to face communication is an important skill to have and I’m not convinced this 
seeming addiction to Facebook is a good thing for our students. 

A Parent’s View of Facebook (D): As a parent of a teenager, there are specific concerns that placate my 
every waking moment and these concerns seem to be growing as my child gets older. Things like the current need to 
drive, relationship woes, academic load, sports, diet, among other necessary teenage wants were all part of the 
parenting package. When my child’s school decided to institute their laptop initiative 3 years ago, all parents and 
students were required to undergo a laptop workshop that dealt with safety concerns, cyberbullying, and the 
technical issues we all would face. However, there was no session regarding the then craze of social network sites. 

The novelty has since withered but not after constant reminders of how much a distraction Facebook had 
become in my child’s life. Every waking moment my child would find some way of getting on to the site, chat with 
classmates who were sometimes sitting across, and Facebook had become the evil of every dinner conversation, 
reprimand, and argument. What it boiled down to was the inability to discipline oneself on time management when 
maintaining a moderate G.P.A., athletic participation and extracurricular activities. 

What was troublesome was more that the school blocked everything else except Facebook. There was no 
monitoring system on student use and it became an issue of treating students like adults and having them decipher 
how to manage their time better. I always thought of the analogy of sticking a donut in a dieting person’s face and 
telling them they can’t eat it. The temptation is there and 13-year-olds were expected to have the discipline enough 
to just say no. 

As said earlier, the novelty is gone, in part to my child’s own understanding that Facebook time does not 
count towards college credits nor is it allowed in most work environments. As a parent, the only positive Facebook 
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attribute is perhaps the social immediacy in knowing that you’re able to meet your friend in 5 minutes down the hall. 
Beyond this notion, it is a big distraction to say the least and has not benefited us in any aspect, added value to our 
family dynamics, or made the bonds of our family stronger. It has kept us updated with birthdays, family photos, and 
gossip but a simple phone call, letter, or email could accomplish this. Perhaps understanding that my child is an only 
child and this was an outlet that pacified the loneliness is up for debate, but I question whether I would allow such 
people into my family to begin with. 

A Parent’s View of Facebook (E): I am a Facebook (FB) user and I have benefited from the strong sense 
of connectedness to current and past associates. The ability to reconnect with past associates is great and I appreciate 
the ways that FB allows me to share my experiences. It is also a great tool for establishing new associations and 
communicating with current contacts. 

As a mature adult I am able to better differentiate between associates and friends. I know that it is possible 
to have many, many associates however friends are rare. I have three sons, one adult and two young adult sons, the 
youngest being 15. I am concerned about their ability to limit their exposure to people who might not have their 
best interest in mind. Also, the amount of Friends in a young user’s account is a status indicator. My 23-year-old 
son has 796 FB friends and my son who just turned 18 has 1,561 FB friends. The 18 year old has always been a 
social child but it is difficult for me to believe that he has and maintains real friendships with 1,561 people. I cannot 
report numbers for my 15-year-old because he has not responded to my friend request. 

The negative effect of “friendship perception” worries me. I question the ability of young people to keep 
their associations in proper perspective. The dangers associated with sharing information with the wrong people can 
have long lasting consequences. Facebook should be more realistic in their use of the term friend. Perhaps they 
should offer users associate as an option. 
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Abstract 

Successful adoption of new teaching and learning technologies in higher education requires the consensus 
of two sub-cultures, namely the technologist sub-culture and the academic sub-culture. This paper examines trends 
in adoption of open source software (OSS) for online learning by comparing the results of a 2009 survey of 285 
Chief Academic Officers and Chief information Officers with the 2006 administration of the same survey. Results 
indicate that while the key drivers of OSS adoption continue to differ for the academic and technologist sub- 
cultures, both sub-cultures converge in deeming total cost of ownership as the most important metric for making a 
go/no-go adoption decision. 



Introduction 

In its sixth annual report on the state of online learning in U.S. higher education, the Sloan Consortium 
reports that more than three quarters of all public institutions in the U.S. are engaged in offering education online 
and that future enrollment growth will be fueled by adults seeking to switch or advance careers in a changing labor 
market (Allen & Seaman, 2008). Additionally, technology expectations of students are forcing institutions to 
improve efficiencies and enhance organizational performance while adopting new technologies to remain 
competitive. One way that organizations have approached technology cost control has been to deploy Open Source 
Software (OSS), software that is distributed with its source code according to the criteria established by the Open 
Source Initiative (Open Source Initiative, 2006). Examples of popular OSS applications include the Linux Operating 
System, Apache for Web servers, and OpenOffice suite. There are also OSS applications for teaching and learning, 
particularly learning management systems (LMSs). The leading OSS LMS products are Moodle 
( http://www.moodle.org ), originally developed in Australia, but currently with a global user base that includes 
nearly 30,000 registerd sites, one million courses, and available to anyone for downloading, and; Sakai 
( http ://www.sakaiproj ect.org ), a platform developed by a group of U.S. institutions that includes generic 
collaboration tools along with teaching and portfolio tools available under an Education Community License. 
Moodle is built on OSS technologies such as PHP, while Sakai is largely Java-based. Other OSS LMS products 
include Claroline ( http://www.claroline.net ), available in more than 35 languages and used in 80 countries; .LRN 
( http://www.dotlrn.com ), a system that has e-commerce and project management applications built in; ATutor 
( http://www.atutor.ca ), developed in Canada and includes more than 17,000 registered user sites, and; Bodington 
( http ://www.bodington.org ), developed in the U.K. and implemented at the University of Leeds and the University 
of Oxford. 

LMSs have become mission-critical services for U.S. colleges and universities. Nearly all (97.5%) U.S. 
institutions of higher education have deployed at least one LMS campus-wide (Green, 2008), enabling them to 
maximize the use of technology investments to support multiple instructional models. Further, more than 3 in 4 
(76.9%) have standardized on a single LMS enterprise-wide, primarily a commercial vendor product (EDUCAUSE 
CORE Data Service, 2007). However, over the past ten years, the number of commercial LMSs has declined from 
several dozen to just a few. This has caused many in higher education to become concerned about a possible 
monopoly in the commercial LMS marketplace. Moreover, proprietary systems do not allow users to modify or 
access the database, making it difficult to produce targeted usage reports, to integrate the system with other campus 
technologies such as student information and financial systems, or to customize the system for a particular campus 
environment (Collins & Committee, 2009). Open source has been touted as a hedge against commercial marketplace 
fluctuations (Lambert, 2005; Villano, 2006; Wheeler, 2007). 

In international education, adoption of OSS teaching and learning applications is relatively mainstream. 
Results of a periodic survey of open and closed source software conducted among more than 450 further and higher 
education institutions in the U.K. (Cornelius, 2006) indicated that the use of open source is on the rise at U.K. 
institutions, with 77% of further and higher educational institutions considering open source in the software 
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selection process and 25% mentioning open source in institutional policy. With respect to survey questions about 
virtual learning environments and LMSs, open source dominates, with Moodle adoption at 39%, followed by 
Blackboard (19%) and WebCT (9%). The 2008 survey indicates an increase in these adoption trends (Canas, 2009). 
Consistent with European goals for free software development, deployment and collaborative research, and the 
European Union’s desire to maintain its lead in the open source arena, the University of Maastricht in The 
Netherlands and the University of Cambridge in the U.K. established a consortium focused on open source projects, 
including the single largest knowledge base on open source usage and development worldwide (Ghosh, 2006). 

In the U.S., however, campus-wide adoption of OSS LMS products is relatively limited, despite the use of 
selected OSS applications by individual faculty or departments (Williams van Rooij, 2007a; Green, 2008). The 
incidence of campus-wide adoption of OSS for teaching and learning can be viewed as a reflection of the divergent 
perspectives of two organizational sub-cultures in higher education: The technologist sub-culture and the academic 
sub-culture. 

Sub-Cultures and Technology Adoption 

Adoption of new technologies requires change, placing pressure on organizational culture - the values, 
symbols, beliefs, stories, heroes, rites and shared assumptions that have special meaning for the organization’s 
employees (Hofstede, 1980; Schein, 1985; Parker, 2000; Hill & Jones, 2001). Organizational cultures are composed 
of discrete sub-cultures or clusters of ideologies, cultural forms and practices, the most distinctive sources of which 
are people’s occupations. Centered around defined, interrelated tasks that create self-definitions and self-perceptions 
as well as perceptions of relationships to other sub-cultures, occupational sub-cultures can serve as potential sources 
of conflict concerning decisions about such issues as the allocation of resources, future goals, changes in practices, 
and criteria used to evaluate performance (Trice & Beyer, 1993). 

Members of the academic sub-culture include faculty, non-technical instructional and research support staff 
(e.g., instructional designers, library staff), and other non-technical staff under the Chief Academic Officer (CAO). 
Although institutional characteristics (Carnegie classification, number of students, public vs. private, for-profit vs. 
non-profit, etc.), culture, discipline, and other factors provide the context in which the academic sub-culture exists, 
concepts basic to this sub-culture include the pursuit and dissemination of knowledge through teaching and research, 
academic honesty, and academic freedom (Umbach, 2007; American Association of Univeristy Professors (AAUP), 
2009). Commitment to these basic concepts means understanding the impact of technology on the processes of 
teaching and learning, on the role of sub-culture members, particularly the faculty, and on how student performance 
is assessed. 

Members of the technologist sub-culture include the institution’s information technology (IT) staff, 
academic computing as well as administrative computing, and the technical instructional and research support staff 
under the Chief Information Officer (CIO). As the pace of technological innovation has increased, the essence of 
this sub-culture, i.e., what it means to be a technologist, is also changing. Keeping abreast of emerging technologies 
means that there is more to think about and process, more perspectives to consider, more complexity to IT as an 
occupation and contributor to the education experience (Alexander, 2009). 

The impact of culture and sub-cultures on technology adoption has been richly explored. In their review of 
the literature on information technology and culture, Leidner and Kayworth (2006) conclude that cultural values 
play a role in determing patterns of technology development, adoption, usage, and outcomes. The occupational sub- 
culture of technologists must be perceived as possessing the knowledge necessary to deploy and maintain a new 
technology. For the academic sub-culture, the ability to capitalize on the maximum learning affordances offered by 
various technologies based on solid pedagogy as well as on awareness of available technologies (Dabbagh and 
Bannan-Ritland, 2005) is a key input to adoption. 

Research Questions 

Examining the effects of the academic and technologist sub-cultures on OSS adoption for online learning 
first requires a clear identification of the penetration of OSS campus-wide. This paper presents the 2009 results of a 
Web-based survey designed to track trends in the adoption of OSS for online learning among U.S. institutions of 
higher education. A baseline administration of the survey was conducted in 2006, a starting point against which to 
measure current and future adoption trends. The specific research questions were: 1) To what extent has the U.S. 
adoption landscape changed over the past three years?; 2) among institutions that have adopted or are considering 
adopting OSS for online learning, what specific applications are being deployed?; 3) what are the key drivers for 
OSS adoption from the perspectives of the academic and technologist sub-cultures?; 4) what policies, processes and 
procedures are institutions putting place to support the adoption of OSS for online learning? 
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Method 



As in 2006, the target respondents for the 2009 survey were Chief Information Officers (CIOs), representing the 
technologist sub-culture, and Chief Academic Officers (CAOs), representing the academic sub-culture (Green, 2008; 
Sheehan, 2008). To obtain results generalizable to the total population of regionally and/or nationally accredited 
U.S. institutions, a list of 450 CAO and 450 CIO names and e-mail addresses drawn via stratified random sampling 
of the various Carnegie classifications was purchased from Higher Education Publications, Inc. (HEP), publisher of 
the Higher Education Directory (Higher Education Directory, 2009). To supplement the pool of potential 
respondents, a call for participation was also posted on the EDUCAUSE web site and targeted those who had 
participated in the 2006 administration of the survey. 

The questionnaire used in 2006 remained largely intact for the 2009 survey administration. Question areas 
included awareness of open source; adoption stages of specific academic and administrative OSS applications; 
reasons for selecting open source; metrics for OSS selection decision-making, and; formal policies and procedures 
around the adoption of new technologies. Consistent with pilot test results used to finalize the survey instrument in 
2006, the survey tool’s SKIP/BRANCH logic was again used so that items related to open source’s impact on 
teaching and learning were asked only of CAOs and items related to the financial impact of open source were asked 
only of CIOs (Williams van Rooij, 2007a). New questions intended to differentiate institutions that are actively 
deploying OSS specifically for teaching and learning versus those deploying OSS for infrastructure (e.g., operating 
systems, databases) were introduced in 2009 based on content reviews by the author’s colleagues in the higher 
education software industry. Further, the software industry-standard scale used to measure the degree of OSS 
adoption of specific applications was expanded to capture any differences at the enterprise level versus the 
unit/department level. For testing the significance of 2006 vs. 2009 adoption, all data was mapped to the original 
2006 scale. To obtain an overall index of the consistency of the revised scales, an item reliability index via 
Cronbach’s Alpha was obtained (Creswell, 2002). The Cronbach’s Alpha coefficient was .934, on a par with that 
obtained in 2006 (.914) and considerably higher than the Social Sciences norm of .700. 

Target respondents were sent a hyperlink to the URL address of a secured third party server hosting the 
survey. Data was collected in the summer of 2009 and the total number of completed surveys was 285. The 2009 
sample is smaller than that obtained in 2006 due to lower response rates (16.1% vs. 28.3% in 2006). This may 
reflect the overall decline in survey response rates among higher education senior administrators over the past few 
years. For example, the 2009 administration of the annual Campus Computing Survey, an industry research project 
with a 19 year history, reports a 10% decline in response rate (Green, 2009). Similarly, response rates for the 2009 
EDUCAUSE Current Issues Survey (Scrivner Agee, Yang, & Committee, 2009) slipped from 2008 and 2007 levels 
(28% vs. 32% and 33% respectively). 

Nevertheless, there is some evidence that response rate is not the primary indicator of survey quality, 
particularly in light of (a) declining response rates across all modes of survey administration (b) studies comparing 
survey estimates to benchmark data from the U.S. Census or large governmental sample surveys, and (c) 
experimental comparisons showing few significant differences between estimates from surveys with low response 
rates and short field periods and surveys with high response rates and long field periods (AAPOR, n.d.). A better 
indicator of whether non-response bias may have affected the 2009 results is the profile of 2009 vs. 2006 
respondents. As shown in Table 1, the 2009 sample demographics are nearly identical to those of 2006. The 
overrepresentation of Doctoral/Research institutions in 2009 is also consistent with 2006, reflecting the first-mover 
role that Doctoral/Research institutions have traditionally assumed in OSS higher education projects, such as Sakai, 
a platform for teaching, learning, and research (The Sakai Project, 2009) and Kuali, an administrative software 
system exclusively for higher education (The Kuali Foundation, 2009). Another check against response bias is the 
extent to which the survey results correspond with other research findings (AAPOR, n.d.). Consequently, the results 
of the 2009 administration of the survey will be compared with other empirical studies as well as with the 2006 
survey results. 
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Table 1. Respondent Demographics, 2006 vs. 2009 





2006 

(n=772) 


2009 

(n=285) 


% Total HEP 
Population 
(2009) 


Carnegie Classification 


Associates 


24.5% 


28.9% 


40.5% 


Baccalaureate and Baccalaureate/Associates 


23.1% 


19.8% 


18.2% 


Masters 


23.8% 


24.7% 


15.7% 


Doctoral/Research 


23.1% 


23.6% 


7.0% 


Specialized Institutions 


4.7% 


3.0% 


17.8% 


Other/Unclassified 


0.8% 


0.0% 


0.5% 


Governance (2008 Digest of Education Statistics) 


Public 


54.3% 


50.2% 


38.7% 


Private, Non-profit 


42.3% 


44.9% 


37.3% 


Private, For-profit 


3.4% 


4.9% 


24.0% 



Summary of Findings 



The Adoption Landscape 

Adoption of OSS for online learning is gaining traction, with dramatic increases in awareness and in 
campus-wide deployment of OSS LMSs. Awareness of OSS is nearly universal (86.4%), increasing slightly from 
the already high levels observed in 2006 (82.6%). This increase is largely due to a rise in CAO awareness levels 
(80.5% vs. 68.5% in 2006), which Ch-Square testing confirms as being significantly higher than in 2006, x2(l, 
n=551)=7.4027, p=.01. To identify potential trends in OSS adoption for teaching and learning versus adoption at the 
technical infrastructure level, CAOs who are aware of OSS were presented with a list of teaching and learning 
applications and technologies and asked to indicate the extent to which their institution has implemented OSS in 
each area using the expanded software industry-standard scale where “1” means “deployed campus-wide”; “2” 
means “deployed in some units/departments”; “3” means “piloting campus-wide”; “4” means “piloting in some 
units/departments”; “5” means “considering campus-wide”; “6” means “considering in some units/departments; “7” 
means “not planned”, and; “8” means “don’t know”, with “2” and “4” being additions to the original 2006 scale. 
Figure 1 illustrates the overall growth in the proportion of CAOs reporting deployment or piloting of OSS. Learning 
management systems (LMSs) show the most dramatic growth (41.1% vs. 22.3% in 2006), which Chi-Square testing 
confirms as statistically significant, x2(5, n=395)=23.830, p=.001. Further, 2009 adoption of OSS LMSs tends to be 
campus-wide rather than at the individual department/unit level. One in four CAOs also reported adopting OSS 
testing/assessment tools (26.8%) and computer-aided instruction (CAI) tools (25.0%), two applications introduced in 
the 2009 survey administration. CIO-reported adoption levels of LMSs (see Figure 2) have increased significantly 
(40.5% vs. 25.8% in 2006), x2(5,n=486)=29.9593, p=.001, as has adoption of digital repositories (32.4% vs. 19.9%), 
X2(5, n=486)=18.5815, p=.01. Adoption of OSS infrastructure applications (computer operating systems, portals) 
remained relatively unchanged. 



241 




45.0% 



Authoring 

Tools 



Online 

Collabora- Student Faculty 

tive Portfolios Portfolios 

Teams CoPs 



200 6(11=271) 20.3% 22.3% 21.3% 

2009(n=124)i 24.2% | 41.1% 27.7% 



Testing 

A ssessm en t C AI To ols 

Tools 



21.3% | 10.0% | 0.09 b | 0.0® o 

l~.9°o ! 1J.4»0 26.8° o 1 25.0° o 



Figure 1. OSS Adoption among CAOs 
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Figure 2. OSS Adoption among CIOs 
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Applications Deployed 

Although Moodle and Sakai are the leading OSS teaching and learning systems in U.S. higher education, 
Carnegie classification is a key differentiator of Sakai adoption. More than half (52.9%) of all 2009 respondents 
stated that they are deploying/planning to deploy OSS teaching and learning applications. When presented with a list 
of OSS teaching and learning applications and asked to identify which of those applications are in active or pending 
deployment, more than 2 in 3 (67.7%) mentioned Moodle, while 1 in 3 (33.0%) mentioned Sakai. Further, Sakai 
deployments were more likely in Doctoral/Research institutions than in institutions in other Carnegie classifications 
(see Figure 3). This finding is consistent with the fact that Sakai was designed as a platform for research as well as 
for teaching and learning. Other OSS teaching and learning applications (e.g., ATutor, ClassWeb, Claroline) each 
generated fewer than 4% of total mentions. 
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Figure 3. Moodle and Sakai Deployment by Carnegie Classification 



Key Adoption Drivers 

Student engagement and the support of active learning continue to be the dominant drivers of OSS adoption 
among CAOs, while CIOs continue to focus on reducing software license fees and on gauging the experiences of 
other institutions who have adopted OSS. Nevertheless, both CAOs and CIOs deemed total cost of ownership as the 
number one metric for making a go/no go decision about OSS adoption. 

Although Chi-square testing showed no significant differences in CAO 2006 vs. 2009 ratings, there were 
some interesting directional changes. In the 2009 survey administration, the strongest positive influence on CAO 
consideration of OSS for teaching and learning was the ability of open source to support engaged learning, to create 
a high-challenge, low-threat environment (47.9% stating “strong positive/positive influence”). This is a modest 
increase from 2006 (43.0%) and slightly displaces the ability to support active learning, to involve students in real- 
world tasks, practice, reinforcement, as the top adoption driver (46.9% stating “strong positive/positive influence” 
vs. 52.6% in 2006). A modest increase in the positive impact of OSS’ ability to support the need to share 
instructional content with other institutions also differentiated the 2009 respondents. At the other end of the 
spectrum, nearly one-third of CAO respondents stated that some of the teaching and learning attributes were not a 
factor in OSS adoption, which is similar to 2006 results. More than one in four (27.8%) stated that a push for open 
source from faculty was a strong positive/positive influence on OSS adoption. 

The largest single influence on CIO consideration of OSS remains the desire to reduce or eliminate 
software license fees, with 3 in 4 (71.0% in 2009 and 75.7% in 2006) CIO respondents stating that this attribute was 
a “strong positive/positive influence”. The second largest influence in 2009 was the experiences of other institutions 
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(48.5%), an increase from 2006 (42.3%) and displacing the need for application functionality as the second largest 
influence (40.0% vs. 47.3% in 2006). Further, Chi-square test results confirmed significant differences between 
2006 and 2009 on four of the ten influence attributes: Need for application functionality not available in commercial 
software, %2(5, n=486)=18.5542, p=.01; Negotiate license agreements with commercial vendors, %2(5, 
n=486)=13.7386, p=.05; Integrate academic & administrative technology services, x2(5, n=486)=34.4826, p=.001; 
Establish/maintain campus-wide standard for software, x2(5, n=486)= 1 8.6723, p=.01. 

As a cross-check on the key drivers of OSS for teaching and learning, respondents in the 2009 survey 
administration were presented with a list of seven institutional metrics and asked to indicate the most important 
metrics that their institutions use to make a go/no go decision on OSS for teaching and learning by ranking each 
metric from “1/most important” to “7/least important”. As illustrated by the mean rankings in Figure 3, the most 
important metrics for making a go/no go decision on adopting OSS for teaching and learning were the total cost of 
ownership (3.03) and faculty satisfaction (3.10), followed by return on investment/value on investment (3.40) and 
student academic performance (3.62). Further, there were no significant differences in mean rankings between CIO 
and CAO rankings. 
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Figure 3. Metrics for Go/No-go Decision-making, 2009 Mean Ratings 
Policies, Processes/Procedures 

CIOs reported great strides in developing formal policies and procedures around the adoption of new 
technologies, particularly in the areas of security (67.0% in 2009 vs. 21.2% in 2006); compliance with the 
appropriate Federal and state regulations (e.g., FERPA, Section 508) (64.8% vs. 14.1%), and ; ownership of 
intellectual property developed by faculty (60.4% vs. 50%). There was almost no difference between CIO and CAO 
responses. 
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Discussion 



The results of the 2009 survey reveal dramatic growth in the campus-wide deployment of OSS LMSs at 
U.S. institutions of higher education. These results are consistent with the latest administration of the Campus 
Computing survey (Green, 2009) , which indicated that OSS LMS penetration has increased from about 5% of all 
U.S. institutions in 2007 to nearly 30% in 2009, with the highest adoption rates among 4-year institutions. 
Nevertheless, Blackboard remains the single campus standard at more than half of the institutions participating in 
the 2009 Campus Computing survey. 

Barriers to campus-wide OSS adoption include (a) the difficulty in calculating the true cost of ownership 
of OSS LMSs, (b) the lack of formal support mechanisms, (c) the need for highly skilled and highly motivated 
technical personnel, (d) the lack of efficient tools for migrating from commercial LMSs, and (e) the lack of 
interoperability with other campus systems (Molina & Committee, 2006; Williams van Rooij, 2007b; EDUCAUSE 
Constituent Group, 2008). Since the original 2006 administration of this survey, however, there has also been some 
discussion of how to overcome those barriers. For example, there are two well-known published guidelines to assist 
institutions in conducting OSS assessments. The Business Readiness Rating (Business Readiness Rating, 2006) 
provides a framework advanced by developers from education and industry to assess the organizational fit of OSS 
based on seven weighted criteria: Functionality, including communication, collaboration, learner assessment, and 
instructional management tools; usability, particularly the ease with which faculty and students can become 
proficient in using the software; the availability and quality of user-maintained documentation for system 
administrators, faculty, and students; the size and activity level of the developer community, as measured by the e- 
mail forums and number of people contributing code; the number and severity of security alerts and the speed with 
which they are addressed; the amount and quality of volunteer and commercial support available, and; the number 
and size of current installations at other institutions. The Open Source Maturity Model (Navica, 2008) is another 
published guideline that enables organizations to self-identify as to how they rate themselves in terms of overall 
maturity in information technology adoption. Based on where they fall in the maturity rankings, organizations then 
assess OSS systems on features/functions, support, documentation, training, product integration, and available 
professional services. 

Individual institutions have also done a better job of placing their own OSS assessment models and 
migration experiences in the public domain (Chao, 2008; O'Laughlin & Borkowski, 2008; Uys & Morton-Alien, 
2007). Live case studies with best practices for selection and implementation have been published by institutions 
from a variety of Carnegie classifications and include information about total cost of ownership of OSS vs. 
commercial software applications, as well as strategies for faculty support (Lakhan & Jhunjhunwala, 2008; Trappier, 
2009; Oakland University, 2009). As more institutions share what they have learned about both the process and the 
impact of OSS selection, OSS adoption for teaching and learning should continue to gain traction. 

Adoption patterns and key drivers as reported by the CAOs and CIO indicate a shift from the strong 
dichotomy seen in the 2006 survey administration to some meeting of the minds in 2009. In 2006, CAO engagement 
with OSS lagged far behind that of CIOs, consistent with what the software engineering literature identified as the 
gap between the technologist who is the end-user of infrastructure-level software and the non-technologist who is 
the end-user of business or academic application-level software, and the need for mutual understanding between 
users and developers (Behlendorf, 1999; Evans, 2002; Glass, 2003; Courant & Griffiths, 2006). Although CAO 
focus remains on technology in the service of pedagogy, the 2009 data indicate that CAOs are beginning to 
recognize total cost of ownership as a critical factor in OSS adoption decision-making. In the same vein, CIOs are 
beginning to recognize the importance of faculty satisfaction and support, along with technical efficiencies and cost 
effectiveness. Consequently, it could be argued that economics is the great equalizer and that the current economic 
climate has pushed cost of ownership into the minds of both academics and technologists (Green, 2009; Claffey, 
2009). In addition, there is now considerable evidence that OSS teaching and learning applications, particularly 
Moodle and Sakai, have evolved into sustainable communities that provide support mechanisms as well as technical 
expertise, reducing traditional barriers to widespread OSS adoption (McDonald, 2009; Collins & Committee, 2009). 
Consequently, the Mellon Foundation’s recent cessation of grant finding to Sakai and other OSS projects is not 
expected to be fatal to Sakai adoption (Parry, 2010). 

Evidence of success is important to the academic sub-culture, particularly for faculty transitioning from 
commercial systems to OSS teaching and learning applications (Sclater, 2008), but also for non-technical support 
staff seeking to build their own best practices inventory. The academic sub-culture responds favorably to OSS for 
teaching and learning when, like any technological change, it is (a) evident, so that there is an awareness of OSS and 
of how OSS is being used, (b) easy to use, without having to choose from a host of features, functions, and complex 
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user interfaces, and (c) essential, so that the what’s-in-it-for-me (WIFM) is clear, rather than being a mandate from 
above (Haymes, 2008). 



Conclusions 

The purpose of this research was to track trends in the adoption of OSS for teaching and learning among 
U.S. institutions of higher education and to explore those trends through the lenses of the academic and technologist 
sub-cultures. Although the research currently has only two waves of measurement - 2006 and 2009 - it offers some 
insights into the pace and patterns of OSS adoption for teaching and learning in the U.S. This suggests opportunities 
for further research in terms of (a) the relationship between the incidence of OSS adoption and the number of 
institutional case studies in the public domain, (b) the continued erosion of adoption barriers through evidence -based 
models that also include data from institutions that utilize third party commercial vendors for OSS implementation 
and maintenance services, as well as institutions utilizing in-house talent and resources, and (c) the pace of OSS 
adoption in the U.S. versus that of the international postsecondary community. Even when the economic 
environment improves, there will continue to be a need to maximize technology investments while providing quality 
postsecondary education. To achieve this requires an ongoing effort to recognize the different perspectives of the 
academic and technologist sub-cultures and striking a balance in which the drivers of one sub-culture are not 
realized at the expense of the other sub-culture. 
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Abstract 

In the digital age, instructional designers must possess both a sound instructional design knowledge base 
and solid project management skills that will enable them to complete courseware projects on time, on budget, and 
in conformance with client expectations. This paper reports the results of a study to identify the extent to which an 
organization’s project management implementation maturity affects roles and responsibilities in educational/training 
product development projects, particularly as regards the instructional design and the project management roles. 
Results show no significant difference by project management maturity level in the roles of instructional designer 
and project manager. However, there appears to be some relationship between maturity level and how organizations 
perceive the skills/competencies of project managers vs. those of instructional designers. Further, organizational 
decision-makers tend to have very specific expectations about the formal education and training of 
educational/training product development project leaders. 

Introduction 

The American Society for Training & Development (ASTD) estimates that U.S. organizations spent $134.1 
billion on employee learning and development in 2008, nearly on a par with the $134.39 billion spent in 2007 
(Paradise & Patel, 2009). These spending levels bode well for educational/training product development projects 
and for the job market for instructional designers. There are many graduate degree programs in the U.S. that prepare 
students for careers in instructional design The standards for instructional design competencies developed by the 
International Board of Standards for Training, Performance, and Instruction (IBSTI) are among the professional 
standards to which advanced degree programs in instructional design adhere, with the ability to plan and manage 
educational/training product development projects listed among the IBSTI advanced competencies for experienced 
instructional designers (International Board of Standards for Training, Performance, and Instruction, 2000). 

Although the instructional design literature is clear about the need to effectively manage 
educational/training product development projects, most models of instructional design view project management as 
embedded within the instructional design process, rather than as a separate methodology (Greer, 1992; Gentry, 

1994; Yang, Moore, & Burton, 1995; McDaniel & Liu, 1996; Stubbs, 2002; Li & Shearer, 2005). However, project 
management is a distinct and evolving discipline, with its own methodology, body of knowledge, and professional 
standards and practices. Further, instructional designer positions require not only instructional design 
skills/competencies, but also project management skills, including the ability to lead a project team, estimate project 
requirements, and develop processes and standards for completion of educational/training product development 
projects. 

Project Management 

The Project Management Institute (PMI), the global standards and credentialing body for the project 
management profession, defines project management as the application of a body of knowledge, skills, tools and 
techniques to project activities to meet project requirements, all of which is documented in the Project Management 
Body of Knowledge (PMBOK® Guide) (Project Management Institute, 2009). Effective project management also 
requires knowledge of the characteristics of the project’s environment (e.g., technology, industry, etc.) as well as 
general management knowledge and skills, and interpersonal skills. Those interpersonal skills - particularly 
communication skills and leadership skills - are deemed essential to today’s successful project manager, where a 
good portion of project activity often takes place in virtual environments (Horine, 2009). Although project 
management has its strongest presence in industries where projects tend to be complex, multi-year and require 
extensive human and financial resources (information technology, construction, etc), project management processes 
and procedures can be used in a variety of industries and for a variety of project types and sizes. 



249 




To assess how much of the project management methodology organizations actually use, Thomas and 
Mullaly (2008) offer a five-level model for assessing the project management maturity of an organization: l)Ad 
hoc, with no organizational implementation of project management; any use of project management processes 
depends on the expertise of individual project managers; 2) some practices, with incomplete or inconsistent 
application enterprise-wide; 3) consistent practices, with a complete project management process in place and 
applied consistently enterprise-wide; 4) integrated practices, with project management as an integral management 
capability that is fully integrated with the organizational lifecycle, and; 5) continually improving practices, with a 
holistic, fully integrated approach to managing projects with a formal and consistently followed process of 
evaluating, assessing, and improving project management implementation. Although the organizations in the 
Thomas and Mullaly study represent a variety of industries and project types, there is no indication as to whether 
any of the projects were educational/training product development projects. 

Educational/Training Product Development and Project Management 

In developing educational/training products, organizations engage in instructional design, a process guided 
by systematic design models and principles focused on establishing and maintaining effective and efficient human 
performance (Rothwell & Kazanas, 2008). The International Board of Standards for Training, Performance, and 
Instruction (IBSTPI) publishes a list of skills and competencies for the instructional design professional, spanning 
novice, experienced and expert designers, with project management listed among the advanced competencies for 
experienced instructional designers. Project management complements the instructional design process by offering a 
set of repeatable processes with which to describe, organize and complete the work required for each phase of the 
project lifecycle, with deliverable complexity also determining how much process is used at each phase (Williams 
van Rooij, 2010). 

Consistent with the success criteria for any type of project, an educational/training product development 
project is deemed successful if it is delivered on time, within budget, and meets the requirements of the project 
stakeholders (Rowe, 2007; Crawford & Pollack, 2007; Horine, 2009; Stubbs, 2002). Even when the roles of 
instructional designer and project manager are filled by the same individual, using project management processes 
enables the educational/training product development project manager to (a) clearly define the project, develop 
realistic schedules, and manage change, (b) choose those processes, levels of detail, and methodology components 
appropriate to the specific project, (c) operate in an organized and efficient manner, and (d) have more time to 
devote to the management “soft" skills, such as team building. However, the ability to apply the specific project 
management processes and knowledge areas are skills that are distinctly different from those of a subject matter 
expert or an instructional designer. When it comes to educational/training product development, the boundaries 
between the instructional design professional and the project management professional are often unclear, with 
conflicting and/or overlapping roles and responsibilities (Dobrovolny, Lamos, Sims, & Spannaus, 2002; Layng, 
1997; Greer, 1992). What is needed is a clear understanding of how these boundaries play out in the real world of 
work. 

Research Questions 

Examining the boundaries between the instructional designer’s role and the project manager’s role requires 
an initial understanding of an organization’s commitment to project management as a distinct methodology. This 
paper reports the results of a Web survey designed to determine the extent to which an organization’s project 
management implementation maturity (PMIM) affects roles/responsibilities in educational/training product 
development projects. The specific research questions were: 1) How much of the project management methodology 
do organizations purport to be using in their educational/training product development projects, as measured by 
Thomas and Mullaly’s (2008) five-level model of project management implementation maturity (PMIM)?; 2) to 
what extent does PMIM affect organizational expectations of an instructional designer’s skill/competencies as set 
down by the International Board of Standards for Training, Performance, and Instruction (IBSTI), versus the 
skills/competencies of a project manager as documented in the Project Management Body of Knowledge 
(PMBOK® Guide) by the Project Management Institute (PMI)? 

Method 

Three segments are essential to the identification of project management maturity in educational/training 
product projects: (a) project management professionals who lead educational/training product development projects 
but are not necessarily instructional designers themselves; (b) instructional designers, and; (c) organizational 
decision makers charged with defining the skills and competencies essential to educational/training product project 
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team leaders (e.g., Chief Learning Officers, Directors of Training and Development). Target respondents were 
recruited via a purposive sampling of members of the Project Management Institute (PMI), the American Society for 
Training and Development (ASTD), and the e-Learning Guild obtained through a call for participation message and 
two reminder messages posted to their respective discussion boards in January - February 2010. 

A total of 1 15 responses were received. However, 12 of those responses were dropped from the analysis 
because the respondent’s organization either did not design/develop educational/training products or the respondent 
did not answer this screening question. Consequently, a total of 103 responses were retained for analysis. About 
one-third (33.7%) of respondents were organizational decision-makers, more than one in four (29.6%) were 
instructional designers, while one in five (21.4%) were project management professionals. The vast majority 
(76.5%) of respondents were in the United States and were employed by companies and departments of various 
sizes, with one in four (24.5%) of those companies in the consulting/contracting sector. There were relatively equal 
percentages of male and female respondents (51.6% and 48.4% respectively), with more than half (57.9%) 
employed in educational/ training product development for 5-15 years. 

The unit of analysis is the respondent because the sampling method (a) enables (but cannot verify) more 
than one individual from an organization to respond and (b) assumes that any respondent is knowledgeable enough 
about his/her organization to respond to the questions about organizational characteristics, including level of project 
management implementation maturity, although this possibility is less worrisome given that one-third (33.7%) of 
respondents are organizational decision makers and as such, are assumed to know what goes on in their respective 
organizations. 

The survey questionnaire used in this research is a composite of the following sources: 1) The self- 
assessment components of the Thomas and Mullaly (2008) study of PMIM that presented respondents with 
definitions of each of the five levels and asked them to select the level that best describes the extent to which project 
management has been implemented in their organization; 2) the professional competencies published by IBSTPI and 
PMI respectively, and; 3) the list of instructional design project manager skills/competencies in Brill, Bishop and 
Walker’s (2006) study of the core success criteria for educational/training product development projects. In 
constructing the survey instrument, generally accepted survey creation guidelines were followed (Wiersma, 2000). 
The questionnaire also included closed-ended demographic questions to obtain information about the characteristics 
of the respondents and of their respective organizations. To identify expected competencies of project managers and 
instructional designers, respondents were presented with a list of IBSTPI and PMI competencies and were asked to 
identify whether each competency is essential for the instructional designer, the project manager, or for both using 
the following 7-point scale adapted from Zimmerman & Kitsantas’ (2005) Perceived Responsibility scale: “1” 
means “always the project manager”; “2” means “mainly the project manager”; “3” means “slightly more the project 
manager”; “4” means “both equally”; “5” means “slightly more the instructional designer”; “6” means “mainly the 
instructional designer”, and; “7” means “always the instructional designer”. 

To assess the instrument’s content validity - a subjective assessment of how appropriate the instrument 
seems to subject matter experts as well as members of the target population (Litchenham & Pfleeger, 2002), a group 
of six experts who have published articles in peer-reviewed journals in the fields of instructional design and project 
management respectively were invited to review the draft survey in October 2009. The most notable outcome of this 
preliminary validation was the recommendation to eliminate the redundancies in the list of project management and 
instructional design competency attributes. The questionnaire was revised to reduce the list of competencies from 38 
to 24, was uploaded to a web hosting service, and the URL posted to the target respondent discussion boards. 

Data validation was conducted to test for the presence of data anomalies such as outliers and missing 
values. Frequency distributions and crosstabs were run to obtain descriptive statistics about respondent 
characteristics as well as project management implementation at the organizational level and at the departmental 
level. The attribute list containing instructional designer and project manager skills/competencies was subjected to 
an item reliability analysis via Cronbach’s Alpha, to measure the extent to which the items are related to each other 
and to obtain an overall index of the internal consistency of the scale (Creswell, 2002). The coefficient for the 
skills/competencies items was .924, which is well above the Social Sciences norm of .700. 

Summary of Findings 



Reported Use of Project Management/PMIM 

The first research question concerns the extent to which organizations purport to be using the project 
management methodology in their educational/training product development projects, as measured by Thomas and 
Mullaly’s (2008) five-level model of project management implementation maturity (PMIM) To that end, 
respondents were presented with Thomas and Mullaly’s (2008) five-level model and were asked to select the level 
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that best describes their organization’s PMIM. Nearly two-thirds (61%) reported low (ad hoc or some practices) 
PMIM levels. When asked to describe the extent to which project management has been implemented in the 
department or function area responsible for educational/training product development, about half (51.3%) described 
those areas as having low PMIM levels. Only about one in four respondents characterized either their organization 
or their educational/training product development department as having a high level of PMIM. 

Looking at PMIM at the departmental/functional level, there are some demographic differences (see Table 
1). Respondents reporting a medium or high PMIM in the departments/functional areas responsible for 
educational/training products were more likely than those reporting low PMIM levels to be responsible for their 
organization’s learning and development strategy, budgeting, and staffing. They tended to be employed in higher 
education, telecommunications, and the non-military sector of government. Their organizations tended to have 500 
or more employees in total, and had 21 or more employees in the educational/training product development area. 
Organizational-level PMIM showed similar differences, although not as pronounced as at the 
departmental/functional area level. 

Table 1 

Selected Sample Demographics by D e p a r t m e n t a I Functional Area PMIM 





Low Maturity 3 (n=56) 


Medium/High Maturity 13 (n=47)* 


Respondent’s Primary Focus 


Responsibility for an enterprise-wide learning & 


24.3% 


44.4% 


development strategy, budget & staffing 


Working on educational/training product 


75.7% 


55.6% 


development projects, either as a team member or 
team leader 
Industry 


Higher education 


2.4% 


20.0% 


Telecommunications 


0% 


8.6% 


Government/non-military 


2.4% 


8.6% 


Company Size 


Less than 100 employees 


28.6% 


17.7% 


101-500 


18.7% 


14.9% 


501-1,000 


17.1% 


15.3% 


1,001-5,000 


21.3% 


32.1% 


More than 5,000 


14.3% 


20.0% 


Area Size 


Less than 5 employees 


37.5% 


22.9% 


5-20 


32.5% 


22.9% 


21-50 


20.4% 


34.3% 


More than 50 


9.6% 


19.9% 



Note. 3 Low Maturity=Ad hoc/level 1, some practices/level 2. b Medium/High Maturity=Consistent practices/level 3, 
integrated practices/level 4, or continually improving practices/level 5. *Caution: Small cell sizes 

Respondents were then asked whether the roles of instructional designer and project manager are fulfilled 
by the same individual or by different individuals. Although 4 in 10 (40.2%) respondents stated that in their 
organization, the instructional designer fulfilled both the roles, there were some differences when looking at the 
responses by project management implementation maturity level. Table 2 shows that as expected, the likelihood of 
the project manager and the instructional designer each fulfilling separate roles was significantly higher (at the 95% 
level of confidence) in organizations characterized by a medium or high level of PMIM in the department/functional 
area responsible for educational/training product development. However, organizations with a medium/high level of 
PMIM appeared to be as likely to have the instructional designer fulfill both roles as they were to have separate 
individuals fulfill those roles. The voluntary comments for medium/high PMIM respondents tended to explain role 
fulfillment according to project size and complexity, while the voluntary comments of low PMIM respondents 
explained role complexity according to the organization’s own business models. 
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Table 2 



Fulfillment of Instructional Designer and Project Manager Roles by Departmental/Functional Area Level PMIM 





Total Respondents (%) 
(N=103) 


Low Maturity 3 (%) 
(n=56) 


Medium/High Maturity 13 
(%) 

(n=47)* 


Instructional Designer 
Fulfills Both Roles 


40.2 




43.9 


36.4 


Project Manager Fulfills 
Both Roles 


11.0 




14.6 


9.1 


Project Manager and 
Instructional Designer 
Each Fulfill Separate 
Roles 


28.0 




17.1 


39.4 


Some Other Arrangement 


20.8 




24.4 


15.1 



Note . 3 Low Maturity=Ad hoc/level 1, some practices/level 2. b Medium/FIigh Maturity=Consistent practices/level 3, 
integrated practices/level 4, or continually improving practices/level 5. *Caution: Small cell sizes 

PMIM and Project Management vs. Instructional Design Skills/Competencies 

The second research question concerns the impact of PMIM on skill/competency expectations. To assess 
the extent to which project management implementation maturity affects the skills/competencies expected of an 
instructional designer versus those of a project manager, respondents were presented with a list of 24 skills and 
competencies and were asked to rate whether each area of expertise is more essential for the project manager, for the 
instructional designer, or for both equally on a 7-point scale: “1” means “always the project manager”; “2” means 
“mainly the project manager”; “3” means “slightly more the project manager”; “4” means “both equally”; “5” means 
“slightly more the instructional designer”; “6” means “mainly the instructional designer”, and; “7” means “always 
the instructional designer”. Looking at the mean ratings in Table 3, it appears that educational/training product 
development departments/functional areas with low PMIM levels were fairly similar to those with medium/high 
PMIM levels in selecting the IBSTPI skills/competencies that are essential for the instructional designer. 
Instructional designers were expected to: Select and use a variety of techniques for determining instructional content 
(Content); select and use a variety of techniques to define and sequence the instructional content and strategies 
(Sequencing); apply fundamental research skills to educational/training product development projects (Research); 
evaluate and assess instruction and its impact (Assessment); select, modify, or create a design and development 
model appropriate for a given educational/training product development project (Design Models); conduct a needs 
assessment to identify the perceived gap between an existing situation and the desired situation (Needs Assessment); 
analyze the characteristics of existing and emerging technologies and their use in an instructional environment 
(Technologies); apply current theory to solve practical problems (Theory). 

At the other end of the spectrum, project managers were expected to possess the following PMI-based 
competencies: Plan, estimate, budget and control costs, so that the project can be completed within the approved 
budget (Costs); organize and manage the project team, so that team member competencies and interactions enhance 
project results (Team Management); define, resource and schedule all activities required to accomplish timely 
project completion (Activities Scheduled); ensure timely and appropriate collection and distribution of project 
information to stakeholders (Information Distribution); identify and manage risks that may impact the project (Risk 
Management); plan and manage educational/training product development projects, including scope, budget, 
schedule, and resources (Project Management); promote collaboration, partnerships, and relationships among the 
participants and stakeholders (Collaboration); define and control what is (not) included in the project (Scope); 
analyze the characteristics of the project environment (Project Environment). Skills/competencies that were 
expected of both the instructional designer and the project manager are: Communicate effectively in visual, oral, and 
written form (Communicate); identify and apply the relevant quality standards, so that results satisfy project 
requirements (Quality); provide for the effective implementation of educational/training products and programs 
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(Implementation); identify and resolve ethical and legal implications of educational/training product development in 
the workplace (Ethical/Legal); make tradeoffs among competing objectives and alternatives (Tradeoffs); anticipate 
and address potential issues before they become critical (Issues). 

Table 3 



Mean Ratings of Project Manager and Instructional Designer Skills/Competencies by Departmental/Functional Area 
PMIM 



Skill/ Competency 


Low 

Maturity 3 

(n=56) 


Medium/High 

Maturity 13 

(n=47)* 


Skill/ 

Competency 


Low Maturity 3 
(n=56) 


Medium/High 

Maturity 13 

(n=47)* 


Content 


5.95 


5.70 


Implementation 


3.81 


3.73 


Sequencing 


5.81 


6.08 


Issues 


3.65 


3.52 


Research 


5.69 


5.37 


Tradeoffs 


3.65 


3.50 


Assessment 


5.68 


5.48 


Project 

environment 


3.51 


3.27 


Design models 


5.51 


5.04 


Scope 


3.14 


3.19 


Target population 


5.50 


4.63 


Collaboration 


2.97 


3.19 


Needs assessment 


5.24 


4.67 


Project 

management 


2.92 


2.54 


Technologies 


5.22 


4.92 


Risk 

management 


2.73 


3.12 


Theory 


4.83 


4.83 


Information 

distribution 


2.73 


2.77 


Communicate 


4.43 


4.63 


Activities 

scheduled 


2.70 


2.73 


Quality 


4.24 


3.88 


Team 

management 


2.65 


2.69 


Ethical/legal 


3.97 


3.80 


Costs 


2.41 


2.08 



Note . a Low Maturity=Ad hoc/level 1, some practices/level 2. b Medium High Maturity=Consistent practices/level 3, 
integrated practices/level 4, or continually improving practices/level 5. *Caution: Small cell sizes 

To drill down on skill/competency expectations, decision-makers - respondents who stated that their role 
focused on responsibility for their organization’s enterprise -wide learning and development strategy, budget, and 
staffing - were presented with a list of formal educational/training alternatives and were asked to select the 
alternative that best describes the type of formal training/education expected of potential leaders of 
educational/training product development project teams. The majority (60.0%) of decision-makers in organizations 
with low PMIM expected project team leaders to have post-graduate certification in Instructional Design, either in 
the form of a Masters degree or a graduate-level certificate from a college or university. Interestingly, nearly one in 
three (30.0%) required no formal education/training. In contrast, decision-makers in organizations with 
medium/high PMIM were less likely to select a “best” source of formal education/training, with one in two (50%) 
offering up their own expectations in the comment box next to the alternative “If other, please specify”. Typical 
comments focused on a preference for both formal and informal knowledge sources. In response to an open-ended 
question about what sources other than formal education were needed, both low and medium/high PMIM 
respondents mentioned experience in managing educational/training product development projects as the means of 
acquiring the knowledge necessary to lead project teams. 
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Lastly, project team leaders - respondents who identified themselves as project managers (PMP or non- 
PMP certified) - were asked how they acquired their knowledge of instructional design and project management 
respectively. Although 10 in 22 project team leaders possessed a formal degree or post-graduate certificate in 
instructional design, nearly all (8 in 10) were in the medium/high PMIM segment. In terms of project management 
knowledge, however, project leaders in both PMIM segments acquired their knowledge from a variety of formal and 
informal sources. Project leaders in the medium/high PMIM segment were more likely than their low PMIM 
counterparts to have had formal project management training. 

Discussion 

This study explored the extent to which organizations that develop educational/training products’s are 
committed to project management as a methodology - as measured by project management implementation maturity 
(PMIM) - that is separate and distinct from the processes of instructional design. The fairly low levels of project 
management implementation maturity (PMIM) observed in this study may reflect the evolving nature of project 
management as a discipline. In a review of the development of project management best practices since 1945, 
Kerzner (2010) notes that project management has become less a series of processes and more of an enterprise-wide 
strategy used to differentiate organizations from their competitors. This relatively new application of project 
management has resulted in organization-specific definitions of best practice which, in turn, makes enterprise-wide 
implementation more complex than just adopting standardized templates and procedures. The churn in applied 
project management practice is also reflected in Thomas and Mullaly’s (2008) finding of great disparity in 
implementation among different organizations and different project types. 

Turner, Ledwith and Kelly (2009) conducted a study of 280 companies to determine the extent to which 
those companies apply project management principles and tools. Study results indicated that regardless of industry, 
the smaller the company, the less robust will be the use of project management processes and tools. The Small 
Business Administration’s criterion for defining a small business is an employee base of 500 or less (U.S. Small 
Business Administration, n.d.). The organizations in the present study that were more likely to have low project 
management implementation maturity levels have fewer than 500 employees, while those with medium or high 
PMIM levels have 500 or more employees. Further, the number of employees in the functional areas responsible for 
educational/training product development in the present study tended to be less than 20 for departments with low 
PMIM levels versus 20 employees or more in departments with medium or high PMIM levels. Consequently, the 
PMIM levels found in this study appear to be in line with what the literature states about the project management 
maturity levels of small versus medium and large firms. In addition, the study finding that education/training 
departments with medium or high PMIM levels tend to be in the telecommunications and government sectors is 
consistent with the historical development of the discipline of project management. A new insight is the finding that 
higher education is also strongly represented among medium/high PMIM level departments. This may reflect the 
fact that higher education is the leading provider of instructional design professionals (GradSchools.com, 2010), 
with departments and functional areas that engage in instructional design projects to support teaching and learning at 
their respective institutions. 

The apparent tendency of respondent organizations with low PMIM levels to have the instructional 
designer fulfill his/her own role plus the role of project manager is also to be expected given the size and relatively 
small number of staff in the organizations. What was not expected was the finding that respondents reporting 
medium or high PMIM levels appear to be as likely to have the instructional designer fulfill both roles as they are to 
have separate individuals fulfill those roles. One explanation may lie in the respondents’ own roles in their 
respective organizations. Respondents in the medium/high PMIM segment were more likely than those in the low 
PMIM segment to include decision-makers responsible for the organization’s learning and development strategy, 
budget and staffing. As such, these respondents would tend to be more knowledgeable about which projects require 
what functional roles. This assumption of knowledge seems to be supported by the voluntary comments in which 
medium/high PMIM decision-makers linked the project leadership role to the size and complexity of a specific 
project. 

The fact that the study revealed no significant differences between the low and medium/high PMIM 
segments on the skills/competencies expected of an instructional designer versus a project manager suggests a 
general consensus as to the expected knowledge base of instructional designers and project managers respectively. A 
more puzzling finding is the apparent disconnect between decision-maker expectations about how candidates for 
leadership of educational/training product development project team should acquire their knowledge of instructional 
design and of project management, and how practicing project team leaders acquire their working knowledge of 
project management and instructional design respectively. On the one hand, decision-makers in both the low PMIM 
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and the medium/high PMIM segments expect team leaders to have an advanced degree in instructional design or 
related field plus solid experience in managing complex educational/training product development projects. 
However, they do not require formal education/training in project management and seem to emphasize “learning by 
doing” as the means of acquiring project management experience. On the other hand, project team leaders reporting 
college/university education/training in instructional design appear to be concentrated in the medium/high PMIM 
segment, while those in the low PMIM segment report relatively limited formal/education in instructional design. 
Further, project team leaders in both PMIM segments have had formal as well as informal training in project 
management. This suggests that practitioners deem formalized project management knowledge as the “ticket to 
entry” for project team leadership, affording them the opportunities and grounding to gain project management 
experience. 

One possible explanation for this disconnect lies in the disciplines that offer formal project management 
education versus those offering formal instructional design education. Project management has its roots in 
production industries and the military, then diffused to other commercial entities and organizations. As such, project 
management is a natural fit with the curricula of degree and certificate programs that educate professionals for those 
industries (e.g.. Business, Engineering). Instructional design programs, however, are usually offered by colleges of 
Education whose long-standing mission has been the education of teachers, the development of education leaders, 
and the advancement of teaching excellence. Although these programs have evolved to capitalize on the teaching 
and learning affordances of technology and prepare students to design instruction in a variety of settings and 
environments, the notion that instruction is a product that requires process management beyond the processes 
provided by instructional design models, is not a good fit with the values, beliefs and shared assumptions of the 
discipline (and sub-disciplines) of Education prevalent in the U.S. (Williams van Rooij, 2010). 

Conclusions 

This study provides insights into the reported usage of project management methodology and perceived 
competencies and roles of project managers and instructional designers in educational/training product development 
project teams.. However, the study does have limitations. The study is constrained by the relatively small number of 
survey respondents. The associations to which the call for participation was issued collectively have nearly two 
million active members representing a diversity of organizations, industries, and geographic regions. As such, the 
results obtained from the survey sample are not generalizable to either the association memberships or to specific 
categories in the data (industry sectors, occupational groups, etc.). In addition to the hypotheses offered in this study, 
there may be alternative hypotheses about why elements of managing projects were embedded into instructional 
design models rather than as separate but complementary processes. Nevertheless, comparison with other studies 
facilitated interpretation of the findings and allowed a richer picture of the observed phenomena to emerge. 

The study also signals opportunities for future research. For example, a scale-up of this study to a larger 
sampling of target respondents would enable an analysis of PMIM by various sample segments. The impact of 
industry, region (including national versus international organizations), and other organizational characteristics on 
PMIM is a research opportunity area. The path to project team leadership of instructional designers with advanced 
degrees and limited/no formal project management training versus those with formal project management training is 
another area that should be tracked. Similarly, the path to team leadership of project managers without advanced 
degrees in instructional design but with formal project management training should be tracked. A comparison of the 
two paths would help determine which combinations of formal/education training take priority in the selection of 
educational/training product development project team leaders. It would also contribute to the exploration of the 
impact of gaps in project management competencies among graduates of instructional design degree and certificate 
programs. 

The findings from this study should contribute to the dialog about how institutions of higher education keep 
the scope of their offerings in instructional design degree and certificate programs targeted toward practititoners 
current with the needs of the job market. By the same token, the impetus for the inclusion of project management 
courses in the instructional design curricula needs to be stressed by the project management community. For 
example, the Project Management Institute (PMI) could do a much better job of advocacy with educational 
institutions, generating awareness beyond the traditional disciplines of business, information technology, etc. PMI 
has begun to take steps in this direction through its educational foundation (Project Management Institute 
Educational Foundation (PMIEF), 2008) offering a project management training fellowship program to assist in- 
service primary and secondary school teachers and administrators in learning the fundamentals of project 
management. The goal of the program is to enable teachers and administrators to utilize project management in 
schools and/or classrooms as a means of enhancing the education of students learning 21st century skills. It is hoped 
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that efforts such as this will contribute to the ongoing dialogue about the relationship between instructional design 
and project management, and that the skills/competencies of the two professions are increasingly seen as 
complementary rather than divergent. 
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Abstract 



Understanding the covert events surrounding the undergraduate university students’ experience is essential to 
educators and counselors involvement in a students’ success. Past research into bullying behaviors has documented 
emotional feelings of anger, sadness and poor concentration as an impact bullying has on the victim. The advent of 
affordable technologies has propagated this concern into cyberspace. This exploratory study was designed to 
evaluate the instances of direct and/or indirect cyberbullying experienced by the undergraduate university student. 
Additionally, the forms of technology utilized in cyberbullying were queried. Surveyors distributed a 27-item survey 
to 120 undergraduate university students in the social science, technology and education departments. The majority 
of all respondents (54%) and one hundred percent of male respondents indicated they knew someone who had been 
cyberbullied. Cell phones, Facebook and instant messaging were the primary forms of technology used to perpetrate 
cyberbullying. Living arrangements and hours of daily technology usage provided statistically significant results. 

The study results provide legitimate concerns regarding the undergraduate university students’ exposure to 
cyberbullying and numerous areas for future research. ( Keywords : cyberbullying, bullying, cyberharrassment, 
undergraduate university students, technology’ and information systems) 
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Growing up, most people have experienced bullying behavior - many times on the playground or on the school bus. 

Negative memories often accompany such reminiscing. Consequently, it seems natural that bullying behavior 
became a research topic, which has evolved with the advent of new technologies. The following section describes 
the research focused on the social and psychological aspects of traditional bullying and serves as a basis for 
understanding technological bullying - cyberbullying in current age, where bullying takes a disturbing twist. 

In the late 1970’s Dan Olweus led social and psychological bullying research and provided a generalized 
understanding of two primary forms direct and relational, or indirect, bullying (Smith & Gross, 2006; Chapel et ah, 
2006). Much of this research focused on students who were either bullies or victims. 

Chapell, et al (2004) explored bullying in university with a sample of 1,025 undergraduates and found that 
24.6 % of respondents had been bullied. Chapell, Hasselman, Kitchin, Lomon, Maclver, & Samllo (2006) found a 
significant positive correlation between being a bully in university, high school and elementary school, with 21 % 
having been bullied. 

Chapell et al. (2006) evaluated 119 undergraduate students to determine the continuance of being a bully, 
victim, or bully-victim from elementary school through university. Interestingly, over 70 % of students who were 
bullied in high school and elementary school bullied others in university. Forty to over 50 % of students who had 
been bully-victims or bullies (respectively) in elementary and high school repeated the pattern in university. 

Victims of bullying struggle with undesirable emotions. They report feelings of anxiety, depression and 
suicidal ideation. Ybarra and Mitchell (2004a) reported an increase of psychosocial problems in those reporting 
cyberbully/victim behaviors including: problem behaviors, drinking alcohol, smoking, depression and low 
commitment to academics. Somewhat alarming are the incidences of school shootings where the perpetrators report 
suffering from bullying behavior. In addition, university counseling centers report increasing concerns of anxiety, 
depression and suicidal ideation with the undergraduate university student (Chapell et al., 2006, p.644). Research 
regarding incidents of bullying on campus is imperative to providing a pro-active approach to the education of the 
21 st Century student. 

The advent of affordable, user-friendly technology has brought bullying into cyberspace. Haber and Haber 
(2007) define cyber-bullying as: 

The use of information and communication technologies such as e-mail, cell phone and pager text 
messages, instant messaging, personal Web sites or blogs and online personal polling Web sites. 
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The technology is used to promote deliberate, repeated and hurtful behavior by an individual or 
group, with the intent to harm others (p. 52). 

This new format of communication has initiated new areas of concern and thus new avenues for analytical research 
and study. 

Past studies of cyberbullying have primarily focused on the adolescent years with emphasis on technologies 
used, reactions to cyber-harassment and the extent of the experience. 

Li (2007) investigated the nature and extent of adolescences’ experience of cyber-bullying. This study 
surveyed 177 students in grade seven of an urban city. The results showed over 25 % of students had been cyber- 
bullied and almost 15 % had cyber-bullied others (Li, 2007). 

A cross-cultural comparison was also undertaken by Li (2008) to determine the extent of and cultural 
differences with bullying noted between students in Canada and those in China. Data from Canada was analyzed 
from two schools in a large western Canadian city with a survey of 157 seventh grade students. The Chinese data, 
from two secondary schools in a large city in Southern China, was gathered from 197 students. In Canadian schools, 
almost 55% of the students were victims of traditional bullying and a quarter had been cyberbullied. Thirty-three 
percent reported bullying others in the traditional form and almost 15% had cyberbullied others. Chinese students 
experienced traditional bullying at a higher rate of 67% while 60% had been cyberbullied. However, when a chi- 
square analysis was used to analyze the cross-cultural occurrences; proportionally, more Canadian students than 
Chinese students reported that they had cyberbullied others (p <0.001) or knew someone being cyberbullied (p = 
0 . 002 ). 

Juvonen and Gross (2008) provided data from an anonymous Web-based survey (conducted in 2005) to 
determine the extent of online bullying experienced by youth between 12- to 17-years-old (N = 1454, mean age = 
15.5, SD = 1.47). Respondents participated via a teen Website (http://bolt.com) and indicated experiences of 
bullying described by the researchers as “ mean things ” students experienced that were defined as “anything that 
someone does that upsets or offends someone else.” There was no requirement of a repetitive nature to the “mean 
thing." Five forms of bullying were reported: insults (66%), threats (27%), sharing embarrassing pictures (18%), 
privacy violation (25%), and password theft (33%). Almost one-fifth of respondents reported repeated experiences 
with 72% of Internet users reporting at least one online bullying encounter during the past year. 
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Two studies addressed aspects of cyberbullying on the university level of students. Finn (2004) found that 
10 to 15 % of 339 students at the University of New Hampshire reported experiencing repeated e-mail or Instant 
Messenger messages that “threatened, insulted, or harassed” and more than half of the students received unwanted 
pornography (surveyed in 2002). 

The psychological needs as a predictor of cyberbullying was also evaluated with surveys distributed to 666 
students in the Faculty of Education at Selcuk University, Turkey. The primary focus of this research was to assess 
the psychological needs of the cyberbully and the results indicated that aggression (“engaging in behaviors which 
attack or hurt others”) and succorance (“soliciting sympathy, affection and emotional support from others”) 
positively predicted cyberbullying, whereas interception (“engaging in attempts to understand one’s own behavior or 
the behavior of others”) negatively predicted it (Kilmac, B., 2009, p. 1313). Additionally, 22.5% of the students 
reported cyberbullying another at least once and 55.3% reported being a victim of cyberbullying at least once in 
their lifetime (Kilmac, 2009). 

The Social Dominance theory may be applied to better understand bullying. Pratto, Sidanius and Levin 
(2006) assessed fifteen years of research that evaluated the social dominance theory. Social groups are divided into 
three classifications: age system , adults have social power over children; a gender system, men have disproportionate 
power compared to women; and an arbitrary-set system, arbitrary groups have access to things of positive and 
negative social value. Arbitrary-set groups may be defined by social distinctions meaningfully related to power. 
Discrimination, that favors dominant groups over subordinate groups, is the primary dynamic that produces these 
group-based social hierarchies. It is the ideologies shared by society, or legitimising myths, that permit this 
discrimination (Pratto, Sidanius, & Levin, 2006). 

Discrimination by individuals is also prevalent. Social dominance orientation (SDO) defines the 
psychological orientations that delineate dominant and subordinate group relations. Although these intergroup 
processes produce better outcomes for dominants than for subordinates, both groups justify their actions and relative 
positions with legitimising myths (Pratto, Sidanius, & Levin, 2006). 

Bullying may be the result of legitimising myths that allow the gender and arbitrary-set systems to 
delineate this power struggle. Technology allows this struggle to exist surreptitiously away from the watchful eye of 
the educator. 
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As technology becomes more accessible to today’s youth more questions arise. CTIA, the International 
Association for the Wireless Telecommunications Industry (2010), indicated that the wireless penetration in the 
United States increased 78% from 1995 to 2009, with 276.6 million wireless subscribers. Subscribers accrue $151.2 
billion in annualized wireless revenues and 1.36 trillion annualized SMS messages. Facebook (2010) has more than 
350 million users with more than 3.5 billion pieces of content (web links, news stories, blog posts, notes, photo 
albums, etc.) shared each week. 

What impact does the inexpensive, readily available cell phone have on the interactions of undergraduate 
students as they maintain their status according to the social dominance theory? How do social networking sites 
impact these students? Has the ability to reply instantly, without personal contact, augmented the bullying scene? 
Additional research is necessary to address the upsurge of technology, and its impact on the age-old events of 
bullying. This study will guide the exploration of the extent of cyberbullying on the university campus to address 
some of these concerns: 

1 . What instances of cyber-bullying does the undergraduate student experience, and what role does gender 
play in cyber-bullying at the undergraduate level? 

2. What forms of technology do undergraduate students use to perpetrate or receive bullying, and how often 
does it occur? 

Method 

Participants 

Department chairs were contacted in four departments at a university in Northeastern Pennsylvania to 
obtain permission to distribute surveys. The selection was designed to diversify the respondents in four divisions of 
study. Ultimately, surveys were distributed in three departments: social science, technology and education. 

A total of 131 students (73 female and 57 male) were surveyed from seven undergraduate classes during 
the 2008 spring semester. Survey questionnaires were distributed to students in class. Students were informed that 
they were not obligated to complete the surveys, and they were instructed to check a box and not complete the 
packet if they had done so in another class. An Information Sheet was provided, for the respondent to keep, that 
defined cyberbullying, as developed by Haber and Haber (2007) and gave the location and hours of operation for 
campus counseling and security. Anonymity was guaranteed with instructions not to record their names on the 
survey; packets were also returned face down in a box provided that was placed away from the surveyor. 
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Measures 



The author adapted and combined surveys from Li (2006) and Spitzberg and Hoobler (2002) to develop a 
27-item survey to address the purpose of this study (see Appendix). Closed-ended questions addressed demographic 
data and asked about instances of hearing about and experiencing cyberbullying at the university where the data was 
collected. An open-ended question was included to allow respondents to offer other instances of cyberbullying 
experienced. 

A pilot study was conducted with twelve undergraduate students ranging in age from 19 to 22. The 
participants reviewed the Information Sheets and completed the study privately. Following completion, a group 
forum was utilized to discuss the Information Sheets, survey format and question specifics. This author obtained 
valuable input that was taken into consideration as the final 27-item survey was created. 

Participants received a six-page packet. The Information Sheet, page one of the packet, contained detailed 
information regarding the purpose of the study, a definition of cyberbullying, and the contact information for the 
university’s counseling and psychological services as well as for campus security. The Haber and Haber (2007) 
definition for cyberbullying was used to introduce the survey. Respondents were directed not to write their names on 
the survey papers and were directed to keep the Information Sheet. 

The second page provided information to the participants regarding the researcher and the purpose of the 
research. Also included was a discussion of the risks, the voluntary nature of the study and informed consent. 

The survey contained a demographic questionnaire to determine gender, age, living arrangements, 
ethnicity, school grade average and hours of technology use per day. When mortalities were removed, data was 
analyzed from 120 students (70 female and 50 male) ranging from age 18 to 24. Of these respondents, two percent 
classified themselves as Asian, two percent Hispanic, seven percent Black, one percent Indian, 85 percent White and 
three percent Other. When queried about living environment, 16 % lived at home with parent/guardian; 53 % lived 
in campus housing; and 3 1 % lived off campus but not at home. School grade averages were reported at 5 1 % in the 
A-B range, 47 % in the B-C range, two percent reported grades in the C-D range and no respondents reported a 
lower than D range. No respondent reported daily use of technology (computer, cell phone, PDA, etc.) below one 
hour daily. Fourteen percent of students use technology between one and two hours daily, 3 1 % between three and 
four hours and 55 % of respondents use technology more than four hours daily. 
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Twenty-one questions were utilized to gather data. Respondents specified their knowledge of students 
being cyberbullied and technologies used. Their direct experience with cyberbullying was analyzed based on 
technologies used, who perpetrated the bullying, the frequency of cyberbullying and whether they told a 
parent/guardian or other adult. The survey was concluded with fourteen specific instances of undesirable and 
obsessive communication via computer or other electronic means. 

Results 

As shown in Table One, 54 % of all respondents indicated knowing someone who has been cyberbullied. 
Of the students that stated they knew someone who had been cyberbullied 50 of the 65 were male; 100 % of male 
respondents knew someone who was cyberbullied. 

Table 1 : Number of Students Reporting Knowing Someone Who Has Been Cyberbullied (N = 

120 ) 
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Specific examples of technologies used to cyberbully were also delineated (Table 2). Of the items listed 
Facebook (56%), Cell phones (45%) and AIM (43%) were the most often reported. 

Table 2: Number of Students Reporting Technologies Being Used to Cyberbully University 
Students (N = 120) 
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Eleven percent of the respondents indicated having experienced cyberbullying at the university (Table 3). 
Of those, Facebook (64%), Cell Phones (43%) and AIM (43%) were the most frequent technologies used. Of the 
respondents who were cyberbtdlied, 57 % were bullied less than four times, 29 % were cyberbullied four to ten 
times and 14 % over 10 times. Students indicated that 50 % of the cyberbullies were university classmates, 57 % 
someone outside of university and 43 % did not know who was cyberbullying them. Seventy-one percent of the 
students replied they had told a parent/guardian or other adult about the cyberbullying incident(s). 
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Table 3: Number of Students Victimized by Cyber'oullying (N = 14) 
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When participants were queried regarding the extent of undesirable and obsessive communication through 
computer or other electronic means (Table 4) four areas were reported at or above 30 %: sending tokens of affection 
(33 %), sending excessively ‘needy’ or demanding messages (30 %), pretending to be someone he or she wasn’t (34 
%) and ‘friending’ people you know to get personal information about you 31 %. An independent-samples t-test was 
conducted to compare these categories. There was no significant difference in scores for males and females. 
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Table 4. Number of students reporting being undesirably and obsessively communicated with or 
pursued through computer or other electronic means (N = 120) 
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Sending excessively explicit messages (e.g. inappropriately giving 
private information about his/her life, body, family hobbies, sexual 
experiences, etc.) 
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Sending excessively ‘needy’ or demanding messages (e.g. 
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A one-way between groups analysis of variance (Table 5) was conducted to explore the impact of living 
environment on whether respondent felt their private reputation had been ‘sabotaged’ via rumors spread with 
computers or other electronic means. Subjects were divided into three groups based on their living arrangements 
(Group 1 : at home with parent/guardian, Group 2: in campus housing, or Group 3 : off campus but not at home). 
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Twenty-three percent (n = 28) of participants answered “yes” to feeling that their private reputations were damaged. 
There was a statistically significant difference at the p < .05 level for the three groups: F (2,1 17) = 3.9, p = .02. The 
effect size, calculated using eta squared, was .06. With only six percent of variability explained by living 
arrangement, future research should address this issue. 

Table 5. ANOVA: Impact of Living Arrangements on 'Sabotage' of Private Reputation 
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Table 6. Tukey HSD Comparison of Living Arrangements and Item K: ‘Sabotage’ of Private 
Reputation 

Post Hoc Tests 
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*. The mean difference is significant at the 0.05 level. 



As noted in Table 6, post-hoc comparisons using the Tukey HSD test indicated the mean score for Group 1 
(M = .32, SD = .478) was significantly different at the p < .05 level (p = .03) than Group 2 (M = .08, SD = .27). 
Group 3 (M = .22, SD = .417) did not differ significantly from either Group 1 or Group 2. The finding that students 
at home with parents/guardians reported a higher incident than those on campus (21.9% and 28% respectively) is an 
interesting result. It could be that students who choose to stay at home feel more susceptible to the comments and 
actions of others. 

A one-way between groups analysis of variance (Table 7) was also conducted to evaluate the effect of 
hours of technology use on the experience of being cyberbullied on the college campus. Subjects were divided into 
four groups based on their technology (computer, cell phone, PDA, etc.) use: Group 1, less than an hour daily (<1 
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hour), Group 2, between one and two hours daily (1 A 2), Group 3, between three and four hours daily (3 A 4) and 
Group 4, more than four hours daily (>4). There was a statistically significant difference at the p < .05 level for the 
four groups: F (2, 117) = 3.16, p = .05. The effect size, calculated using eta squared, was .05. With only five percent 
of variability explained future research is needed. 

Table 7. ANOVA: Effect of Hours of Technology Use on Being Cyberbullicd 
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As noted in Table 8, post hoc comparisons using the Tukey HSD test indicated that the mean score for 
Group 3 (M = .03, SD = .164) was significantly different at the p < .05 level (p = .049) that Group 4 (M = .18 SD = 
.389). Groups 1 and 2 did not differ significantly from either group 3 or 4. Recently, an undergraduate college 
student stated that he was “online practically 24 hours straight” during a snowstorm. Future research should analyze 
this in more depth to better understand what >4 hours entails. 



Table 8. Tukey HSD Comparison of Individual Being Cyberbullied and Hours of Computer Use 



Post Hoc Tests 
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When students were given the opportunity, in an open-ended question, to express individual experiences 
with being undesirably pursued through computer or other electronic means, three students responded. One recalled 
the following event: 
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I have a friend who was dating a boy named Jim on my space for an entire year. She met Jim though a 
friend on the basketball team we will call LL. For a while it was going very well with my friend and Jim. 
Then he started getting obsessive. He was never home when she tried to see him. Finally, after a year and 
being in love with Jim she finds out that Jim and LL are the same person. She was heartbroken because LL 
created the perfect guy and is a girl. So, my friend was harassed by LL for a long while and then the cops 
were involved. LL sought help. In the end, my friend lost her senior year to false promises and hope. 

Additionally, a respondent discussed a random person who signed in to a chat room and pretended to be the 

respondent to harass their friends. This occurred on several occasions before the person quit. A third participant 

stated they were “rick rolled.” Defined by Wikipedia (2010) as being directed to a Web link unrelated to the topic 

being discussed. The Web link actually takes the user to the Rick Astley music video, “ Never Gonna Give you Up. ” 

Discussion 

This study explores a little examined area of the undergraduate experience. The dearth of literature in this 
area left this researcher with only two similar studies to evaluate. The preliminary analysis of the 120 participants 
indicated that the majority (54%) of respondents knew someone who had been cyberbullied. One hundred percent of 
the males surveyed indicated they knew someone who had been cyberbullied. 

Eleven percent had personally experienced cyberbullying, with 57 % having been cyberbullied less than 
four times, 29 % four to ten times and 14 % over ten times. Finn’s (2004) findings support the results since 10 - 15 
% of his respondents received email or Instant Messaging that “threatened, insulted, or harassed.” 

When specific examples of incidents of undesirable and obsessive communication through the computer or 
other electronic means were queried 33 % of respondents had received unsolicited tokens of affection, 34 % had 
someone pretend to be someone he or she wasn’t and 30 % experienced excessively ‘needy’ or demanding 
messages. Twenty-three % of respondents had received pornographic or obscene images. This is relatively lower 
than Finn’s (2004) finding of 58.7%. Finn did not query if the pornography was sent directly to respondent or via 
group messages. 

The findings bring to light a discrepancy. Interestingly, 30% or more students indicated that they had 
experienced incidents of undesirable and obsessive communication. Based on the operational definition of 
cyberbullying that was given in the student survey, the incidents of undesirable and obsessive communication are 
forms of cyberbullying. One would expect that all of those who indicated that they had experienced these incidents 
would have also reported yes to cyberbullying. Instead, only 1 1% reported “yes” to being cyberbullied, instead of 
the expected 30%. One questions why these undesirable incidents are higher than the 1 1 % that responded yes to 
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being cyberbullied. Is it possible that these instances are not considered bullying but are accepted as a part of social 
life “online?” 

From the data, 71 % of the respondents indicated that they had told a parent/guardian or other adult about 
the cyberbullying experience. Upon reflection, the researcher questions whether a respondent considers a university 
peer to be an “other adult?” 

To better understand who was doing the cyberbullying, question four queried if the respondent was 
cyberbullied by classmates (50 %), someone outside of university (57 %) and “I don’t know” (43 %). The combined 
percentage of greater than 100 % brings the question as to whether the victims were being bullied by more than one 
person. 

A statistically significant difference was noted in two areas of the study. There is a 95 % confidence that 
the apparent difference in students who felt their private reputation had been ‘sabotaged’ among students living at 
home with parent/guardian or in campus housing is a genuine one and not a sampling error. A lower percentage of 
students who lived on campus reported feeling ‘sabotaged’ (21.9%), than those living at home with parent/guardian 
(28%). Twenty percent of students living off campus but not at home felt their reputation had been damaged. 

The second area of research that indicated a statistical significance indicates a 95 percent confidence that 
the apparent difference in students that were cyberbullied among students who used technology between three and 
four hours daily and those on technology more that four hours daily is also genuine and not a sampling error. Those 
students who used technology more than four hours daily reported more incidents of being cyberbullied. 

When studying traditional bullying at the university level, the findings produce smaller results than this 
research. Chapell, et al (2004) explored bullying in university with a sample of 1,025 undergraduates and found that 
24.6 % of respondents had been bullied. Other research, conducted by Chapell, et al (2006) found a significant 
positive correlation between being a bully in university, high school and elementary school, with 21 % having been 
bullied. Unlike traditional bullying, this exploratory research indicates an increase of bullying, via a cyberbullying 
format, to up to 34 %. The advent of affordable technology and thus an increase in technology usage over the past 
several years may contribute to this increase. 

Students report feeling angry, sad and hurt when cyberbullied. Poor concentration and low school 
achievement is also a concern (Beran and Li, 2005, p. 270). Chapell et al. (2006) report on a number of studies that 
found that most school shooters had been bullied. This supports a concern that further campus violence is possible 
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due to this relatively new form of bullying, via technology. University educators and counseling centers need to be 
aware of the ability of undergraduate students to surreptitiously and repeatedly bully victims via technology. 

Pratto, Sidanius and Levin (2006) evaluation of the Social Dominance Orientation (SDO) provides a basis 
of understanding for the cyberbullying actions of the undergraduate student. In the stressful environment of 
university, students may feel they must dominate to succeed. The relative ability to feel control of a situation as the 
dominant bully may allow the cyberbully to justify their actions. Kilmac (2009) supported this theory with the 
finding that aggression and succorance positively predict cyberbullying. 

This researcher faced several limitations. Data gathered from 120 participants inhibits the ability to provide 
overall generalizations at the university being studied. Also, it is not possible to generalize to a national level. The 
convenience sample format of this exploratory research may not have provided a widely diversified study group. 
Finally, standardization of the surveying procedure was not possible since the researcher did not administer the 
survey to every group. 

Future research is needed to expand our understanding of cyberbullying at the university level. One area of 
research would further analyze the undergraduate university student’s concept of cyberbullying versus the 
undesirable and obsessive communication through computer or other electronic means. More detailed gender data 
would be valuable to determine if the males who responded they knew someone that had been cyberbullied were 
referring to men or women. Are they more often the confidant of fellow female students? In addition, more 
information needs to be gathered to determine if one or more perpetrators are victimizing those being cyberbullied. 
When one considers the relatively high percentage of respondents who told a parent/guardian or other adult about 
being cyberbullied, it would be valuable to know if these respondents are considering university peers as the “other 
adult.” Finally, a nation-wide survey of undergraduate students would provide valuable data of cyberbullying 
experienced. 

Conclusion 

To the best of my knowledge, this study is the first to closely examine the extent of cyberbullying that 
undergraduate students experience. It provides a basis for future research in cyberbullying on the undergraduate 
campus. The ability of individuals to surreptitiously bully others via technology combined with results indicating 
54% of respondents knowing someone who had been cyberbullied and up to 34% having been cyberbullied indicates 
a need for such research. 
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The academic college setting values the mature, eschewing the sophomoric behavior of bullying, but even 
the university is not safeguarded from cyberbullying in a technological age. Those concerned with the welfare of 
students need to be abreast of their cyber troubles. 
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Appendix 



INFORMATION SHEET 
(Please read before starting) 

PLEASE DO NOT WRITE YOUR NAME ON THE SURVEY PAPERS. 

As part of my MCOM research study, I am looking at what characteristics of cyber harassment students at 
University experience. The information in this packet will help me in that area and will be kept confidential. I 
appreciate your help, but if you do not wish to participate, you can return an empty packet. 

Cyberbullying is defined by Haber and Haber (2007) as the use of information and communication technologies 
such as e-mail, cell phone and pager text messages, instant messaging, personal Web sites or blogs and online 
personal polling Web sites. The technology is used to promote deliberate, repeated and hurtful behavior by an 
individual or group with the intent to harm others (p. 52). This may include a course of conduct, with repeated 
actions over time and an invasion of a person’s relative right to personal privacy (Spitzberg & Hoobler, 2002: 73). 
This survey, being conducted to fulfill graduate requirements at University, does not qualify me to help with issues 
of cyberbullying. The following services are available at THE UNIVERSITY : 

□ University Counseling and Psychological Services, often called CAPS, is available to assist you with 
family issues, depression, anxiety, relationship difficulties, loneliness, eating issues, alcohol and drug 
concerns, career decisions and more. The office is open Monday through Friday 8:00 AM to 4:30 PM. 

□ The University has made a commitment to ensure the safety of its students, employees and visitors. 
Through the implementation of safety procedures, policies and educational programs, the university has 
enhanced the awareness of safety throughout the campus. Campus security is located at . . .. 

When you are finished with this survey, please place it in the box at the front of the room with the print side down. 
If you have any questions about this study, I can be contacted at cmw8410@ esu .edu. 

This In formation Sheet is yours to keep. Thank you for your time © 
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February 2008 



Cyber Flarassment and the Undergraduate University Student 

You are being invited to participate in a research study about cyber harassment and its impact on university students. 
I am conducting this study from the Media Communications and Technology department, at University, under the 
supervision of my professor. This study is being conducted in fulfillment of a graduate student project for the 
completion of a master degree in education. 

There are no known risks if you decide to participate in this research study. There is no cost to you for participating 
in the study. The information you provide will be used to compile data regarding the characteristics of cyber 
harassment at University. The questionnaire will take about twenty minutes to complete. The information collected 
may not benefit you directly, but the information learned in this study should provide more general benefits. 

This survey is anonymous. Do not write your name on the survey. No one will be able to identify you or your 
answers, and no one will know whether or not you participated in this study. Individuals from the Institutional 
Review Board may inspect these records. Should the data be published, no individual information will be disclosed. 

Your participation in this study is voluntary. By completing and placing this survey in the collection box on the front 
desk, you are voluntarily agreeing to participate. You are free to decline to answer any particular question you do 
not wish to answer for any reason. I hope you will take a few minutes to complete this survey. 

If you have any questions about the study, please contact .... 

The Institutional Review Board has reviewed my request to conduct this project. If you have any concerns about 
your rights in this study, please contact Dr 



Date of IRB Approval: March 2008 
IRB Number: 

Project Expiration Date: March 2009 
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I I PLEASE CHECK THIS BOX AND DO NOT COMPLETE THE SURVEY IF YOU HAVE ALREADY 
PARTICIPATED IN ANOTHER CLASS. 

□ Please answer each of the following questions as accurately as you can. 

□ For those in which multiple responses are provided, you need only fill in the appropriate response(s) 
for you. 

□ For the others, please place your answer in the blank space provided for each. 

1 . I am a female male. 

2. My age is, please check one: 18-24 25 & above 

3. During this university semester I am living: Please check one 
At home with parent/guardian 

In campus housing 

Off campus but not at home 

4. How do you describe yourself, please check one: 

Asian Hispanic Black 

Indian White Other 

5. My school grade average is usually, please check one: 

A - B range B - C range C - D range 

Lower that D range 

6. I use technology (computer, cell phone, PDA, etc.), please check one: 

Less than an hour a day 

Between 1 and 2 hours a day 

Between 3 and 4 hours a day 

More than 4 hours a day 

1 understand that completion of this survey implies informed consent and voluntary participation. 

1) Have you heard of students at University using technology to bully/harass other students, if yes please 
check all that apply: 

E-Mail 

Cell phones [text, pictures, video or messages] 

Video cameras, Web cam 

AIM 

Facebook 

My Space 
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Blogging 
Twitter 
Chat Rooms 



2) I have experienced cyberbullying at THE UNIVERSITY (e.g. email, cell phones, video/Web cams, 
AIM, Facebook, My Space, Blogging, Twitter, Chat Rooms, other) If no; please go to question #6. 

Yes 

No 

3) If yes, I experienced cyberbullying via (check all that apply): 

E-Mail 

Cell phones [text, pictures, video or messages] 

Video cameras, Web cam 

AIM 

Facebook 

My Space 

Blogging 

Twitter 

Chat Rooms 

Other: please explain 



4) If yes, I was cyberbullied by (check all that apply): 

University classmates 

Someone outside of university 

I don’t know 



5) If yes, 1 have been cyberbullied, please check one: 

Less than 4 times 

4-10 times 

Over 10 times 



6) I know someone who has been cyberbullied: 

Yes 

No 

7) When I was cyberbullied, I told a parent/guardian/or other adult about it: 

Yes 

No 

Has anyone ever undesirably and obsessively communicated with or pursued you through computer or other 
electronic means by (CHECK ALL THAT APPLY): 



a. Sending tokens of affection (e.g. poetry, songs, electronic greetings, praise, etc.) 

b. Sending exaggerated messages of affection (e.g. expressions of affections implying a more 

intimate relationship than you actually have, etc.) 

c. Sending excessively explicit messages (e.g. inappropriately giving private information about 
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his/her life, body, family hobbies, sexual experiences, etc.) 



Sending excessively ‘needy’ or demanding messages (e.g. pressuring to see you, assertively 
requesting you to go out on a date, arguing with you to give him/her ‘another chance,’ etc. 

Sending pomographic/obscene images or messages (e.g. photographs or cartoons of nude 
people, or people or animals engaging in sexual acts, etc.) 

Sending threatening written messages (e.g. suggesting harming you, your property, family, 
friends, etc.) 

Sending sexually harassing messages (e.g. describing hypothetical sexual acts between you, 
making sexually demeaning remarks, etc.) 

Sending threatening pictures or images (e.g. images of actual or implied mutilation, blood, 
dismemberment, property destruction, etc.) 

Exposing private information about you to others (e.g. sending e-mail out to others regarding 
your secrets, embarrassing information, unlisted numbers, etc.) 

Pretending to be someone he or she wasn’t (e.g. falsely representing him/ herself as a 
different person or gender, claiming a false identity, status or position, pretending to be you, 
etc.) 

‘Sabotaging’ your private reputation (e.g. spreading rumors about you, your relationships or 
activities with friends, family, partner, etc.) 

‘Sabotaging’ your work/school reputation (e.g. spreading rumors about you, your 
relationships or activities in organizational networks, electronic bulletin boards, etc.) 

‘Friended’ people you know to get personal information about you 

Other: Please explain 
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Survey adapted from: 

Li, Qing. (2006). Cyberbullying in schools: a research of gender differences [Electronic Version]. School 
Psychology’ International, 28, 157- 170. 

Spitzberg, Brian H. & Hoobler, Gregory. (2002). Cyberstalking and the technologies of interpersonal terrorism 
[Electronic Version]. New Media & Society, 4, 71 - 92. 
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Abstract 

The purpose of this content analysis study was to investigate how mixed methods were used by researchers 
in the Instructional Design and Technology (IDT) field. This study reviewed multi-data studies, studies that collected 
both quantitative and qualitative data, from three major journals in the IDT field. The results indicated that 
multi-data studies were rare in IDT literature. In addition, only 64.4% of the selected multi-data studies were mixed 
methods studies, while the rest of them were multi-method studies. Amongst the mixed methods studies, the 
embedded and triangulation designs with the emphasis on the quantitative data were preferred by IDT researchers. 

Introduction 

The increasing importance of the mixed methods research across the social sciences has turned more and 
more researchers into this comparatively new research methodology (Clark, Huddleston-Casas, Churchill, Green, & 
Garrett, 2008; Creswell, 2003; Hanson, Creswell, Clark, Petska, & Creswell, 2005). In mixed methods studies, both 
qualitative and quantitative data are collected, analyzed, and mixed in a same research project instead of 
emphasizing the contrasting paradigmatic features between the qualitative and quantitative methods. Researchers 
suggested that the qualitative and quantitative method has different strengths and bias; therefore, integrating these 
two methods has the possibility of combining the strength from both research methods while reducing the weakness 
of either approach (Creswell & Plano-Clark, 2007). For example, the strength of qualitative methods lay at its thick 
and rich description of a focused phenomenon, while quantitative methods are valued for their ability to find the 
relationship between phenomena to provide generalizable results. Following up a quantitative dominated study with 
a comparatively small qualitative data to increase the depth of the study’s findings is one of the approaches to make 
use of the combined effects of qualitative and quantitative methods. 

Education research, as a sub-branch of social sciences research, is inherently complex and multi-faceted. The 
learning processes are exposed to the impact of the delivery mode, the interaction between learners and the 
instructor, the group dynamic, the learning environment, the learners’ characteristics, and the design of the content. 
The Instructional Design and Technology (IDT) field is an academic field that concerns itself with these issues 
through the instructional designs. It has great impact on the integration of instructional technologies into instruction 
and the development of appropriate instructional strategies for effective learning. IDT is also believed to hold the 
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new era of e-learning (Woodall, 2004), which requires the consideration of more variables for effective learning. 
Mixed methods research has great promise to improve the understanding of the complex phenomena in integrating 
technologies into instruction and explorations of instructional designs by providing researchers with both 
quantitative and qualitative perspectives. 

Albeit the beneficial of mixed methods research, there is sparse literature in the IDT field that employed 
mixed methods research design. The discussion about the mixed methods studies is difficult to locate, and little has 
been written about how the mixed methods designs were applied in the IDT field. This study sought to investigate 
how mixed methods studies were used by researchers in the IDT field through investigating journal articles from 
three major IDT journals. This study reviewed the concepts of mixed methods research; stated the study purpose and 
scope; investigated the nature of mixed methods IDT research; and discussed the findings and the future 
applications. 



Overview of Mixed methods Research 
Mixed Methods and Multi-method Research 

Hanson and his colleagues (Hanson, et al., 2005) pointed out that the use of multiple data collection methods 
could date back to the earliest social science research. Under the same using- multiple -data- collection-methods 
canopy (Tashakkori & Teddlie, 2003b), mixed methods and multi-method research are terms that are often used 
interchangeably by some researchers (e. g. Clark, et ah, 2008). However, they are not necessarily the same. As the 
mixed methods research gradually became the third research paradigm in educational research (Johnson & 
Onwuegbuzie, 2004), it is necessary to differentiate it from multi-method research (Tashakkori & Teddlie, 2003b). 

Multi-method research, as defined by Hunter and Brewer (2003), is “the practice of employing different 
types, or styles, of data-collecting methods within the same study or research program” (p. 577). This definition 
indicates that multi-method studies include studies that employ both quantitative and qualitative data collection 
methods, or multiple forms of either qualitative or quantitative methods (Creswell & Plano-Clark, 2007). 

Mixed methods research, on the other hand, has more specific qualifications. There are several versions of 
definitions of mixed methods research. Mixed methods research was defined as “the collection or analysis of both 
quantitative and qualitative data in a single study in which the data are collected concurrently or sequentially, are 
given priority, and involve the integration of the data at one or more stages in the process of research ” (Creswell, 
Clark, Gutmann, & Hanson, 2003, p. 212). Greene, Caracelli, and Graham (1989)defined it as a design “that include 
at least one quantitative method (designed to collect numbers) and one qualitative method (designed to collect 
words), where neither type of method is inherently linked to any particular inquiry paradigm ” (p.257). Tashakkori 
and Teddlie (2003a) defined mixed methods as a research design that combine qualitative and quantitative 
approaches into the methodology of sing-or-multi-phased research study. The mixed methods research was also 
defined as “a research design with philosophical assumptions as well as methods of inquiry.” “As a method, it 
focuses on collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series of 
studies” (Creswell & Plano-Clark, 2007, p. 5). Although these definitions have slightly different focuses and details, 
they share the same emphasis on collecting, analyzing, and mixing both quantitative and qualitative data for 
conducting a mixed methods study. In sum, although multi-method and mixed methods research may both collect 
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quantitative and qualitative data, multi-method research does not have to mix the data as mixed methods research 
does. 

As suggested by Hartley and Morphew (2008), content analysis requires a clear measure of judgment. In 
order to serve this purpose, this content analysis study used Creswell and Plano-Clark’s (2007) definition to identify 
mixed methods studies. That is to say, only when a study that collected, analyzed, and mixed both quantitative and 
qualitative data within the same study or research project would be identified as mixed methods study. According to 
this definition, studies in which the researcher collected only one type of data, but used “mixed methods way of data 
analysis” would not be counted as mixed methods studies. For example, in a study that the researcher only collected 
qualitative data and transformed it into quantitative data by counting, coding, or developing themes would not be 
counted as mixed methods research in this content analysis study. In addition, two types of studies that fall into the 
category of multi-method would be not counted as mixed methods studies either. First, multi-method studies that 
used multiple quantitative or qualitative data- collecting methods would not be taken as mixed methods study. 
Second, multi-method studies that collect and analyze both quantitative and qualitative data without mixing them 
would not be counted as mixed methods studies either. 

When a particular study was only a part of a serial study, it was difficult to found out how the entire serial 
study was conducted without sufficient information. For that reason, this specific content analysis study focused 
only on single studies. In sum, mixed methods research is different to multi-method research. One of the goals of 
this study was to differentiate the mixed methods studies from the multi-methods studies in the pool of selected 
research articles. 

Characteristics of a Mixed Methods Study 

Hanson, Creswell, Clark, Petska, and Creswell (2005) suggested that other than three traditional steps, 
designing a mixed methods study involved three additional steps. The three traditional steps are deciding on the 
purpose of the study, the research questions, and the type of data to collect. The three additional steps include 
deciding whether to use an explicit theoretical lens, identifying the data collection procedures, and identifying the 
data analysis and integration procedure. The theoretical lens refers to the theories that underline a researcher’s study 
and subsequent methodological choices. The second step is about deciding on data collection procedure and 
priorities. To be specifically, this step decides on whether the quantitative and qualitative data will be collected 
concurrently or sequentially, and whether the emphasis will be put on one method or put equally on both methods. 
The third step involves deciding the point at which data analysis and data mixing will occur. Step two and step three 
resonate with the three characteristics of mixed methods research that was suggested by Creswell and Clark (2007): 
timing, style, and priority. In the following section, these three characteristics will be discussed through the 
introduction of the two mixed methods designing steps. 

Data collection timing, design type, and priority. According to Creswell and Clark (2007), there are two 
types of data collection procedure or timing: concurrent and sequential. The four mixed methods research designs 
suggested by Creswell and Clark were triangulation, embedded, explanatory, and exploratory design. Any one of 
these four designs can fit into these two procedures. The exploratory and explanatory use sequential data collection 
procedure while triangulation uses concurrent data collection procedure. Unlike the other three designs they can 
only use one data collection procedure, the embedded design, however, can use both concurrent and sequential data 
collection procedure depend on each specific study. 
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In a triangulation design, the qualitative and quantitative methods are implemented during the same 
timeframe with equal weight. The data were mixed during the interpretation or analysis stages. In an embedded 
design, the qualitative method is a small complementary portion of the overall quantitative design or vice versa. The 
exploratory design is a two-phase design, in which the qualitative data is often used to develop a quantitative 
instalment or taxonomy. The explanatory design is also a two-phase mixed methods design, in which the qualitative 
data helps explain or build upon initial quantitative results. The explanatory design has two design models: follow- 
up explanation model and participant selection model. In the participant selection model, the quantitative 
information is collected to identify and purposefully select participants for a follow-up, in-depth, qualitative study. 
In embedded, explanatory, and exploratory design, priority is usually unequal and gives to only one type of data, 
qualitative or quantitative data. This specific content analysis study used triangulation research design. 

Points of data analysis and data mixing. In mixed methods studies, data analysis and integration may occur 
by analyzing the data separately, by transforming, or by connecting them depending on the type of design (Creswell 
& Plano-Clark, 2007; Hanson, et al., 2005). In the concurrent approach, the quantitative and qualitative data 
analyses are separated, while the data are merged during the interpretation or analysis. In the sequential approach, 
the quantitative and qualitative data are connected in which the analysis from the first phase of the research is used 
to guide data collection in the second phase. 

The quantitative section of this content analysis focused mainly on the mixed methods designing step two 
and three suggested by Hanson, Creswell, Clark, Petska, and Creswell (2005). This study first used the data mixing 
as the key criterion to differentiate the mixed methods studies from the multi-methods studies. After that, this study 
looked into the data collection procedure, the design type, and the priority in the mixed methods studies. The 
qualitative section of this study focused on investigating the rationales provided by researchers for collecting both 
quantitative and qualitative data. 

Advantages of Mixed Methods Research Design 

Researchers have pointed out various advantages that mixed methods research design could bring into a 
research project. For example, Johnson and Onwuegbuzie (2004) suggested that the “mixed methods research offers 
great promise for practicing researchers who would like to see methodologists describe and develop techniques that 
are closer to what researchers actually use in practice” (p. 15). Because all methods have inherent biases and 
limitations, mixed methods design may offset or counteract these biases and limitations (Greene, et al., 1989). 
Mixed methods research have the possibility to bridge the schism between quantitative and qualitative research 
(Onwuegbuzie & Leech, 2005), offer insights that could not be gleaned from quantitative or qualitative approach 
alone (Bryman, 2007), provide a better understanding of research problems (Creswell, 2005), and clarify the nature 
of research intentions for researchers and writers (Bryman, 2006). Despite the advantages of mixed methods 
research design, the mixed methods research was rarely used in the IDT research. Among 1,134 articles from three 
major IDT journals from 2004 to 2009, the researcher only foundl4 (1.2%) of them used both qualitative and 
quantitative methods without differentiating mixed methods and multi-method studies. This preliminary result 
indicated the necessity and importance of future investigation on the use of mixed methods research design in the 
IDT field. Guided by the advantages of mixed methods design, this content analysis study chose to use a mixed 
methods design. 
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Research Purposes and Research Questions 



The purpose of this content analysis study was to investigate how mixed methods studies were used by 
researchers in the IDT field A concurrent triangulation mixed methods design was used for this study in order to 
“validate and expand quantitative results with qualitative data” (Creswell & Plano-Clark, 2007, p. 62). The 
quantitative questions were used to find out the characteristics of selected mixed methods studies. Concurrent with 
this data collection, the qualitative analysis explored the rationales that were provided by researchers for using both 
qualitative and quantitative methods. The reason for collecting and mixing both quantitative and qualitative data was 
to bring together the strengths of both forms of research methods in order to have a more complete view of the 
research questions and findings. 

The merging of quantitative and qualitative data happened at all stages of this study, including the design, 
data collection, data analysis, and inteipretation stages. In the results phase, a mixing table was used to demonstrate 
the interrelationship between the research type (mixed methods or multi-method research) and the provided 
rationales for using both quantitative and qualitative data. In order to find out how mixed methods studies were 
conducted and reported in the IDT field, this study reviewed three first tier IDT journals guided by the following 
research questions: 

Quantitative : What was the type of the study that collected both quantitative and qualitative data in IDT 
literature, multi-method or mixed methods study? 

Quantitative : What were the characteristics of mixed methods studies in IDT literature in terms of their data 
collection timing, design type, and priority? 

Qualitative : What were the rationales for collecting both qualitative and quantitative data in IDT literature? 
Mixed'. How did emerged rationales for collecting both qualitative and quantitative data related with the 
type of study in IDT literature? 



Research Methods 

Content analysis is an empirical method for examining text and images in order to identify messages and 
meanings (Krippendorff, 2004), which is popular in the social sciences research (Hartley & Morphew, 2008). In 
order to find out how mixed methods studies were conducted and reported in the IDT field, this study first identified 
whether a single study was a mixed methods study or a multi-methods study when it employed both qualitative and 
quantitative methods. After that, this study investigated the characteristics of mixed methods studies in terms of their 
data collection procedure, design type, and priority according to the definition and categorization suggested by 
Creswell and his colleagues (Creswell, 2003; Creswell, et ah, 2003; Creswell & Plano-Clark, 2007). At last, this 
study summarized the rationales provided by researchers for using both qualitative and quantitative methods in the 
mixed methods studies and the multi-methods studies. The qualitative analysis included coding of themes of authors’ 
rationales for using both quantitative and qualitative methods. Mixed analyses included merging the quantitative and 
qualitative data during the analysis and interpretation. 

Sampling 

Fourteen research articles were selected from three major IDT journals: Educational Technology’ Research 
and Development (ETRD), Journal of Research on Technology’ in Education (JRTE), and TechTrends: Link Research 
& Practice to Improve Learning (TT). These three journals are first-tier IDT journals suggested by instructors in 
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Virginia Tech. This study included articles that were published in these three journals from January 2004 to 
November 2009. Year 2009 was truncated to exclude December since the search was conducted in November of 
2009. The researcher used the EBSCOhost Education Databases to locate the journals. 

The researcher searched titles and abstracts for the following logic operators and terms: mixed methods, 
multi-method, [quantitative AND qualitative], or [survey OR questionnaire OR experimental) AND (focus group OR 
interview)]. Table 1 shows the criteria that were used to identify whether a study is a mixed-method study or a 
multi-method study. A study was identified as a mixed method study only when it collected, analyzed, and mixed 
both quantitative and qualitative data in a single study based on Creswell and Plano Clark’s (2007) “methods 
definition” (p. 12) of mixed methods research. 

Table 1 



Criteria for Identifying a Mixed Methods Study 



Collection 


Analysis 


Mixing 


Single Study 


Study Type 


Yes 


Yes 


Yes 


Yes 


Mixed methods Study 


Yes 


Yes 


No 


Yes 


Multi-methods Study 


Yes 


No 


Yes 


Yes 


Multi-methods Study 


No 


Yes 


Yes 


Yes 


Exclude 


Yes 


Yes 


Yes 


No 


Exclude 



A summary sheet was developed and used to collect both quantitative data and qualitative data by the 
researcher. The summary sheet included three sections. The first section dealt with the article’s basic information 
such as title, year of publication, and journal name. Section II included four close- ended quantitative research 
questions. The first research question was used to identify the research type of a study: a mixed methods or 
multi-method study. When identified as a mixed methods studies, the three following questions were used to identify 
the studies’ data collection procedure/timing, design type, and priority. Section III focused on the qualitative 
research question about the rationales that were provided by researchers for using both qualitative and quantitative 
methods. 



Results 

This research search located a total of 20 articles with the following hits in each journal: ETRD (4), JRTE (8), 
and TT (8). From this pool, 6 articles were excluded as they did not meet the inclusion criteria for several reasons. 
These articles were excluded because they were conceptual articles in which qualitative and quantitative data 
collection were only discussed without the conduction of data collection. The final sample consisted of 14 empirical 
studies with the following distributions in each journal: ETRD (4), JRTE (7), and TT (3) (see Appendix A). In 2004, 
none of the three journals published studies that collected both quantitative and qualitative data. JRET has most 
evenly distributed publication of the studies that collected both quantitative and qualitative data while JRET’s 
publication of the studies was clustered in 2008. 

Quantitative Research Question 1 : Mixed methods or Multi-method Studies 

Among the selected 14 articles, 9 articles (64.4%) were mixed methods studies, and 5 articles (35.7%) were 
multi-method studies. In addition, 50% of the selected articles (n=7) either had “mixed methods” in their titles or 
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indicated using mixed methods directly in the article. About 86% (n=6) of the studies that had stated the intention to 
conduct a mixed methods study were actually mixed methods studies. In other words, about 67% of the mixed 
methods studies in IDT literature had clear indication of conducting a mixed methods study. The study found that 
the majority of articles from ETRD and JRTE were mixed methods studies in which 75% of the ETRD articles (n=3), 
71% of the JRTE articles (n=5), and 33% of TT articles (n=l) were mixed methods studies. The two multi-method 
studies from TT reported the results from one research method, though both quantitative and qualitative data were 
collected. 

Quantitative Research Question 2: Characteristics of the Mixed methods Studies 

This study investigated the time, type, and priority of the nine mixed methods studies. As shown in table 2, 
among the nine mixed methods studies, six of them collected qualitative and quantitative data concurrently, and 
three of them collected data in a sequential manor. In other words, more researchers in IDT field preferred 



concurrent mixed methods design. 
Table 2 

Data Collection Procedure 






Time 






Frequency 




Percent 


Valid Percent 


Cumulative Percent 


Valid concurrent 


6 


66.7 


66.7 


66.7 


sequential 


3 


33.3 


33.3 


100.0 


Total 


9 


100.0 


100.0 





As indicated in Table 3, 55.6 % of studies (n=5) used embedded mixed methods design; 33.3% of studies 
(n=3) used triangulation design; and 11.1% of studies (n=l) used explanatory design. None of the mixed methods 
study used exploratory design. In addition, most studies that used the embedded design conducted experimental 



studies and used qualitative data for supplementary purposes. 
Table 3 

Mixed Methods Design Type 






Type 








Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid triangulation 


3 


33.3 


33.3 


33.3 


embedded 


5 


55.6 


55.6 


88.9 


explanatory 


1 


11.1 


11.1 


100.0 


Total 


9 


100.0 


100.0 





As shown in table 4, no study put the priority on qualitative data. About 66.7% of studies (n=6) put more 
weight on quantitative data while 33.3% of studies (n=3) put equal weight on both quantitative and qualitative data. 
In addition, no single mixed methods study emphasized qualitative data over quantitative data in the IDT literature. 
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Table 4 



Priority on Qualitative or Quantitative Methods 





Frequency 


Priority 

Percent 


Valid Percent 


Cumulative Percent 


Valid quantitative 


6 


66.7 


66.7 


66.7 


equal 


3 


33.3 


33.3 


100.0 


Total 


9 


100.0 


100.0 





Qualitative Research Question: Reasons for Using Both Methods 

The analysis of the content found that multi-method studies tended to provide only general rationales or even 
no rationales for collecting both qualitative and quantitative data (n=3). Among the five multi-method studies, three 
of them did not provide reasons for collecting both quantitative and quantitative data. The other two studies only 
provided reasons such as “though understanding” or have a “more complete picture” of the study. Other than that, 
the qualitative data were also used to compensate quantitative data when quantitative data found not significance 
(Mouza, 2008). 

On the other hand, mixed-method studies tended to provide more specific reasons for employing both 
quantitative and quantitative data-collecting methods. As to the embedded design the rationales provided by 
researchers suggested the supplemental role of the qualitative data collection to “corroborate”, “better explain ”, 
“help”, “compensate”, and “supplement” the quantitative findings or methods. Studies that used triangulation, in 
general, emphasized the characteristic of the study design or the requirement of research questions as the reasons of 
employing both quantitative and qualitative methods. In addition, their description of the rationales were more likely 
to place the equal weight on both quantitative and quantitative data by using word such as “cross-validate”, or 
bidirectional statement such as “Quantitative data were gathered to answer quantitative research questions; 
qualitative data helped supplement and explain the quantitative findings” (Williams, Ma, Prejean, & Ford, 2007, p. 
206). The first column of table 5 lists the themes of authors’ rationales for collecting both quantitative and 
qualitative data. 

Table 5 presents the number of each themes used by mixed methods or multi-method studies. If an article’s 
rationales fit into two themes, then it would be counted into the final number of each theme. For example, if an 
article statement the reason for collecting both quantities and qualitative data was to use qualitative data to validate 
quantitative data and to provide better understanding of the results, in this case, number “1” will be added to both 
themes. From this table, we can see, using one data set to “corroborate” or “validate” another data set was the most 
popular rationales f n — 4) provided by mixed methods studies in IDT literature while “improve depth” and 
“minimized errors” were the least used rationales (n=l). 
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Table 5 



Reasons for Collecting both Quantitative and Qualitative Data 



' — — — __^^^Type of Study 

Themes — — — . . 


Mixed methods Studies 
(n=9, 64.3%) 


Multi-method Studies 
(n=5, 35.7%) 


Corroborate/validate 


n=4 


n=l 


Supplement 


n=2 




Compensate 


n=3 




Provide better understanding of results 


n=2 


n=l 


Improve the breadth 


n=3 


n=l 


Improve depth 


n=l 




Minimize errors 


n=l 




Decide by the natural of the study 


n=3 




No reason 




n=3 



Discussion 

This content analysis study was conducted with a purpose to investigate how mixed methods design was 
used by researchers in the IDT field. The results indicated that both mixed methods and multi-method studies were 
still rare in IDT research. The lack of mixed methods and multi-method studies may attribute to the eight barriers 
suggested by Bryman (2007), such as researchers’ faith in one method, publication issues, and so on. 

Results from the quantitative data indicated that 64.4% of studies (n=9) were mixed methods studies in IDT 
literature. About 67% of the mixed methods studies specified themselves as mixed methods studies in their titles or 
in the content. It implies that most IDT researchers, who collected both quantitative and qualitative data for their 
studies, considered the mixing of different data sets necessary for conducting a mixed method study and knew how 
to mix data. 

The quantitative data indicated that among the four mixed methods designs suggested by Creswell (Creswell, 
2003; Creswell, et ah, 2003; Creswell & Plano-Clark, 2007), the concurrent embedded (55.6 %) and triangulation 
design (33.3%) were the most popular design in IDT research while the exploratory design was the least popular one. 
The preference of concurrent design may due to the research design, time management, or researchers’ data 
collection preference. Within the limitation of this content analysis, the reason why the concurrent data collection 
approach was more popular in IDT studies was not explored. The reason of the popularity of the embedded mixed 
method design in IDT literature might be the reasons that was concluded by Mathison (1988) “the experimentally 
inclined are enjoined to use qualitative research methods to help conceptualize their studies” (p. 13). The popular of 
triangulation design might due to the researchers’ desire to use multiple methods, data sources, to enhance the 
validity of research findings (Mathison, 1988). These might be the possible reasons why researchers in IDT field 
preferred to use triangulation and embedded mixed methods design. However, these assumptions are not proved by 
scientific research. Therefore, in order to find out why these two designs were preferred by IDT researchers, a 
separate study is required. 

Generally, mixed methods studies tended to emphasize the quantitative data in IDT literature. In studies that 
used embedded design, the qualitative data tended to take a secondary role to complement the quantitative data. 
Because the priority and design are closely related, the reference of the embedded experimental mixed methods 
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design may explain why most mixed methods studies put more weight on the quantitative data instead of qualitative 
data. In addition, since triangulation design normally put equal emphasis on both quantitative and qualitative design, 
it also explains why the number of studies that had equal emphasis on both qualitative and quantitative data was 
same to the number of the studies used triangulation design. 

The qualitative data suggested that IDT researchers who conducted mixed methods studies tended to 
provided more specific reasons for collecting both quantitative and qualitative data than researchers who conducted 
multi-method studies. Some multi-method studies provided no reasons regarding to collecting both quantitative and 
qualitative data. It may imply that mixed methods researchers had careful study design in terms of the data 
collection, data analysis, and data use. Contrary to the careful design of mixed methods studies, the design of 
multi-method studies might be more based on the researchers’ hunch. Moreover, the qualitative data indicated that 
the most frequently used reason for collecting both quantitative and qualitative data in IDT literature was using the 
qualitative data to valid and compensate quantitative data. In addition, researchers also believed that using both 
quantitative and qualitative data could help them to get a better understanding of the nature of their study. 

Conclusion 

The purpose of this study was to investigate how mixed methods studies were conducted in the IDT field. 
With this purpose in mind, three most popular journals in the IDT were selected; and studies that collective both 
quantitative and qualitative data from 2004 to 2009 reported in these three journals were located and investigated. 
From 1134 articles, only 14 articles collected both quantitative and qualitative data in a single study and 9 of them 
were identified as mixed methods studies. The small amount of the mixed methods study indicated that albeit the 
advantages of mixed methods design, the mixed methods studies were still rare in the IDT field. The findings of this 
study indicated that the IDT researchers preferred concurrent and triangulation mixed methods design. The findings 
also indicated that most researchers who conducted mixed methods research (66%) had intention to conduct a mixed 
methods study by indicating it directly in the titles or in the articles. In addition, this study also found researchers 
who conducted mixed methods studies tended to provide more specific reasons for collecting both qualitative and 
quantitative data than researchers who conducted multi-method studies. 

The findings of this study suggested confliction between the advantages of the mixed methods research and 
the lack of it in the IDT field. By helping IDT researchers to be aware of this confliction, this study may encourage 
more IDT researchers to get to know and conduct more mixed methods research. In addition, this study can also help 
IDT researchers to be aware of the important components of a mixed methods study, such as deciding the timing for 
mixing quantitative data and qualitative data and clarifying reasons for mixing. At last, this study can also be 
considered as a sample for conducting a mixed methods study. 

In addition to the academic contributions, this study had several limitations. First of all, because this study 
only traced back to empirical studies from 2004, it limited the generalizability of the findings. Second, this study 
only investigated studies reported in three IDT journals; therefore, it may not be broad enough to represent the entire 
IDT field. Last, the author was the only researcher who searched the articles, selected the articles, and judged 
whether a study was a mixed methods study or a multi-method study. It is possible that the author missed some 
articles during the article search process, as well as mislabeled articles in the data analysis phase. For the future 
research, the researcher triangulation, in which multiple researchers are involved in the data collection, analysis, and 
present is expected to enhance the validity of research findings (Mathison, 1988). 
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In the future, a study that investigates the characteristics of mixed methods studies such as the frequency of 

mixed methods studies spreading over time, the data collection instalments, and the research topics can contribute to 

the understanding of mixed methods studies in the IDT field. In addition, the specific reasons why researchers 

choose to use a certain mixed methods design can also be explored in the future. 
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Examining Instructional Design Principles Applied by Experienced Designers 

in Practice 

Cindy S. York & Peggy A. Ertmer 

For many years, there has been an ongoing conversation about how to improve instructional design (ID) 
education (Bennett, 2010; Larson & Lockee, 2009; Shambaugh & Magliaro, 2001; Silber, 2007). This conversation 
has included questions regarding the efficacy of teaching ID models to novice designers, based on the fact that, in 
practice, models are applied neither consistently nor uniformly (Wedman & Tessmer, 1993; Dick, 1996). For 
example, Wedman and Tessmer determined that practitioners frequently omitted recommended ID activities from 
their projects. While some have argued that we should continue to teach models to novice designers due to the 
foundational knowledge they provide (Dick, 1996), others believe we should be teaching relevant skills, such as 
problem solving, instead (Jonassen, 2008). 

If ID models are ineffective in helping novices learn how to practice ID, what alternative approaches are 
available? Three of the more common recommendations include contextualizing ID by presenting students with 
real-world design situations (Hartt & Rossett, 2000), teaching problem solving strategies students can use to solve 
real design problems (Jonassen, 2008), and using case -based instruction to engage students, vicariously, in the 
realities of practice (Bennett, 2010; Ertmer & Quinn, 2007; Ertmer & Russell, 1995). 

The first method involves presenting instructional design students with real-world contexts in order to 
develop the skills necessary to function as competent designers (Dabbagh & Blijd, 2010; Hartt & Rossett, 2000). For 
example, Dabbagh and Blijd used a situated and problem-based environment in which they engaged students in a 
team-based, authentic instructional design problem for a real client. Hartt and Rossett (2000) also paired students 
with real organizations to act as instructional design consultants. This allowed students to experience the daily 
activities of an ID practitioner such as interacting with the client, understanding project management, and working 
on a team to solve real problems. 

A second suggestion for improving ID education is to teach students how to solve ill-structured problems, 
as opposed to how to follow step-by-step procedures, as suggested by ID models (Jonassen, 2008). Jonassen argued 
that learning how to solve ill-structured problems leads to more effective designers. In addition, Jonassen argued 
that because design problem solving comprises a cyclical decision-making process, students should be taught how to 
meet constraints and client satisfaction while simultaneously solving the problem. Silber (2007) supported this 
notion when he argued that ID is comprised of a set of principles, or heuristics, used by designers. 

A third recommendation for teaching instructional design is case-based learning (Bennett, 2010). Case- 
based learning (CBL) consists of students working on instructional design cases, presented through text or 
multimedia (Bennett, 2010), to discuss, analyze, reflect on, and make recommendations for solving the presented 
design problems. CBL can promote decision making by requiring students to reflect on the experience while 
explaining, justifying, and critiquing decisions made during the process (Bennett, 2010). Like situated learning, 

CBL utilizes authentic design problems, but in a safer environment where students can make mistakes without dire 
consequences. Similarly, Jonassen and Hernandez-Serrano (2002) proposed using practitioners’ stories to help 
situate design problems, thereby enabling students to better understand the ill-structured nature of real design 
problems. 

Applying ID Models in Practice 

The conversation on improving ID education has included debate over whether, and to what extent, 
practitioners use ID models in their practice (Jonassen, 2008; Silber, 2007; Wedman & Tessmer, 1993). Wedman 
and Tessmer surveyed 73 practicing instructional designers to determine which parts of the ID models they used in 
their work. Results showed that designers selectively chose which ID activities to complete or omit and that only 1 
of the 73 participants reported completing every activity for every design project on which they worked. 

The use of ID models has been recommended and supported by a number of textbooks in the field (Dick, 
Carey, & Carey, 2008; Smith & Ragan, 2005; etc), as well as by personnel in the military (Branson, 1977; Finch, 
1987), and industry (Branson, 1977). However, Jonassen (2008) stated that adhering to a process model seldom 
leads to successful outcomes. Silber (2007) agreed when he stated, “teaching the ADDIE model step-by-step. . . does 
NOT provide learners with either complete mental models of ID principles or sets of heuristics . . . that can be 
selected and modified based on the uniqueness of the ill-structured problem presented in each situation” (p. 11). In 
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1996, Dick countered that his text was written for beginners and was never intended to describe “what practitioners 
actually do” (p. 58). Still, for novice learners, the model offers a workable strategy for how to approach design 
problems encountered in practice (Dick, 1996). According to Dreyfus and Dreyfus (1986), novices must first follow 
learned rules until they have accumulated enough experience, at which point they can set aside those rules and 
proceed using their past experiences to guide them. 

Yet, experts in the field have described a number of problems related to relying on ID models to solve 
authentic design problems. For example, Gordon and Zemke (2000) stated that although “every training designer is 
schooled in some version of it,” (p. 43), the ADDIE model should be abandoned for a number of reasons: (a) it is 
“too slow and clumsy to meet today’s training challenges” (p. 44), (b) when “used as directed, it produces bad 
solutions” (p. 44), and (c) it is better at describing what excellent designers “do, but [is] not very useful for doing 
those things” (p. 48). 

Silber (2007) suggested that instructional designers do not actually follow procedural ID models. Rather, 
they engage in problem solving throughout the ID process, using principles derived from a multitude of sources: 
standard basic textbooks, different ID philosophies, research, and commonly accepted design practices. In place of a 
procedural model, Silber proposed “The Silber ID Principle-based Model” to represent a mental model used by an 
expert instructional designer (p. 11). Silber’s model is based on principles derived from the four aforementioned 
sources and is intended to be considered and used, as needed, as designers “define and solve ID problems” (p. 10). 
However, Jonassen (2008) argued, “teaching students a set of principles and heuristics, specifically if done in the 
absence of context, will not help students learn to make decisions” (p. 26). 

To what extent do the mental models of ID experts include the principles proposed by Silber? How, 
exactly, do experts apply their ID knowledge, including their knowledge of ID models, when solving complex 
problems of practice? In order to answer these questions, we turned to the literature describing the practice of expert 
instructional designers. 

Examining Expert Practice 

In the field of instructional design, advancement has been made in defining expertise (Ertmer & Stepich, 
2005; Eseryel, 2006; LeMaistre, 1998), including delineating the differences between novices and experts when 
solving ill-structured design problems (Ertmer et ah, 2009; Ertmer et ah, 2008; Eseryel, 2006; LeMaistre, 1998; 
Nelson, 1988; Perez & Emery, 1995; Rowland, 1992). For example, Rowland (1992) found that experts interpret, 
analyze, and represent problems differently than novices. Experts generated a list of potential solutions (versus 
establishing firm solutions), which were different than those proposed by novices. In addition, experts retrieved and 
utilized different resources and considered different and additional factors than novices when solving the design 
problem. Rowland proposed that some of these differences could be related to the mental models used by the 
experts. 

In a study by Perez and Emery (1995), five experts and four novice designers were provided with a 
troubleshooting problem and asked to think aloud in order to examine their design problem-solving processes. In 
their results, Perez and Emery reported that experts and novices used different types of knowledge; that is, whereas 
novices used theoretical knowledge (such as ID models), experts also used strategic knowledge, which was based on 
experience. In a follow-up article on the same study, Perez, Johnson, and Emery (1995) described how their expert 
instructional designers used ID principles during the design process. For example, one of their participants 
“suggested that his practice was to treat theoretical principles as heuristics” (p. 340). This is similar to what 
Romiszowski reported in 1981: experienced designers used principles, based on the ID models learned in school, to 
solve ill-structured problems. 

More recently, Ertmer et al. (2008) described how the seven experts in their study used organized 
collections of domain-specific knowledge, acquired through years of experience, to solve an ill-structured design 
problem. One of the main findings from this study was that the experts narrowed the problem space and interpreted 
the problem situation using specific, and unique, frames of reference, built from their knowledge and experiences. 

Frames of Reference 

Although personal frames of reference have been described in the literature, the specifics related to these 
frames of reference are unknown; although we have a general idea of where they come from (knowledge and 
experience), we do not know the principles or models that comprise them. 

In an attempt to answer these questions, Visscher-Voerman (1999) conducted a study with 24 experienced 
designers to determine the design strategies used by professional, high-reputation designers in various training and 
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education contexts. According to Visscher-Voerman, one of the factors that influenced the design process and 
solution was the designer’s frame of reference, which was comprised of their experiences, perspectives, and ideas 
from previous projects on which they had worked. Other researchers, in describing how experts work, have used 
similar terms; for example, Dreyfus and Dreyfus (1986) referred to this as perspective ; Bransford, Brown, and 
Cocking (2003) described how experts organized their thinking around the core concepts or big ideas in the field; 
while Walker (1971) referred to experts’ frames of reference as a platform of ideas (p. 52), which he described as 
including ideas about what is, as well as what ought to be. According to Walker, this platform, then, guides the 
curriculum developer [designer] in determining “what he should do to realize his vision” (p. 52). Wilson (1997) 
noted similarities between personal theories and mental schemas, stating that they both “provide a framework for 
acting intelligently” (p. 22). Based on the work of Kelly, Wilson defined frames of reference as “personal theories 
that guide our perceptions and actions in what would otherwise be an impossibly confusing world” (cited in Wilson, 

p. 22). 

In the Ertmer et al. (2008) study, the frames of references used by the seven experts to solve the given ID 
problem were somewhat unique from each other and seemed to be related to their current roles in the field. In 
addition, during problem analysis, all of the experts incorporated a mental model of the instructional design process 
within their frames of reference. This was verified in a subsequent study (Ertmer, York, & Gedik, 2009), in which 
we analyzed the stories of 16 “seasoned” professionals who described complex or challenging ID projects on which 
they had worked. We determined that the vast majority of our participants applied an adapted or modified ID model 
when analyzing the problem. We concluded that experts use instructional design models, although not necessarily in 
the manner described in ID textbooks. 

An additional finding from this research was the existence of rules of thumb, or a set of guiding principles, 
used by the participants. These rules of thumb were found to be relevant to the individuals telling the stories, but 
also included universal elements across participants. However, given the limited scope of this research (i.e., 
relatively few participants, primarily representing academia), we felt that these ‘shared’ principles needed 
verification from a larger participant population, working in a wider variety of contexts. 

Therefore, the overarching purpose of this research was: (1) to examine and describe the principles that 
guided instructional designers’ practice and (2) to identify the extent to which participants’ frames of reference 
included components of the instructional design model. 

Method 

The current study was designed to examine the ID principles used by experienced instructional designers 
during the design process. The Delphi technique (Linstone & Turoff, 1975) was used to present the guiding 
principles identified in the 2009 study (York, Ertmer, & Gedik, 2009) to a panel of experienced instructional 
designers (n=31). Successive Delphi rounds were conducted until panel consensus was reached. 

Delphi Process 

The Delphi process in this study comprised 10 steps: (1) selection of participants, (2) creation and review 
of the first Delphi survey, (3) execution of the first Delphi survey to present principles from the 2009 study and 
provide an open-ended forum for new possible principles, (4) data analysis of Round I responses, (5) creation and 
review of the second Delphi survey, (6) execution of the second Delphi survey, (7) data analysis of Round II 
responses, (8) creation and review of the third Delphi survey, (9) execution of the third and final Delphi survey to 
determine stability or consensus of principles, and (10) final analysis and dissemination of Delphi results to 
participants. 

Selection of Participants 

An initial email was sent to 54 instructional designers requesting their participation in a study about 
principles used by experienced instructional designers during the design problem-solving process. This initial list 
was generated by the researchers and included the names of designers known to be experienced in the field. 
However, only 14 responded. Therefore, a request was posted on Linkedln.com asking for experienced instructional 
designers to participate by completing a demographic survey. An additional 80 people responded to the survey for a 
total of 94 responses to the demographic survey. The demographic survey requested the following information: 
name, email, gender, age range, current position and title, formal education, summary of instructional design 
background, and instructional delivery formats they currently use in instructional design. From the demographic 
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survey, a purposeful sample (Patton, 1990) of the 50 most experienced instructional designers was selected 
according to criteria published in the instructional design and expertise literature (Eseryel, 2006; LeMaistre, 1998; 
Perez et al., 1995; Rowland, 1992). This criteria included the following, in order of importance: (1) minimum of 10 
years of hands-on experience, (2) currently practicing ID, (3) number and level of educational degrees, (4) 
nominated or recognized by peers, (5) diverse experiences, (6) on-going training/education/certification, and (7) 
manager/trainer (for apprentice instructional designers). 

The 50 selected participants were emailed an invitation to participate on the Delphi panel. Of those 
emailed, 35 responded as willing to participate. For the Round I survey, 35 participated, however, from Round 1 to 
Round II, 4 dropped out of the study. This left a panel of 3 1 participants who completed all three surveys. 

The final panel membership consisted of 18 females (58%) and 13 males (42%). Ages were listed by range: 
5 were in the 31-40 range (16%), 14 were in the 41-50 range (45%), 10 were in the 51-60 range (32%), and 2 were 
in the 61+ range (7%). The panel had an average of 19.7 years of experience with instructional design, ranging from 
10 to 43 years. 

Delphi Process and Timeline 

The three Delphi rounds were conducted over a 2-month period in summer 2009. Surveys were provided 
online, hosted on a secure server. An email was sent to participants describing the Delphi procedure as well as how 
to access the surveys. Follow-up emails were sent to participants if they did not respond to a survey during the 2- 
week open period. Successive rounds of surveys were initiated by an email to the participants informing them the 
survey was open and requesting their responses within 2 weeks. Once consensus was reached, an email was sent to 
inform participants that the Delphi rounds had ended, to thank them for their participation, and to request their 
possible participation during the verification of the results. 

Each Delphi survey was reviewed by a five-member team before being presented to the Delphi participants. 
The review team consisted of an experienced instructional designer (Ph.D.) currently practicing at a higher 
education institution, two professors of educational technology, and two Ph.D. students who had completed all 
required coursework in instructional design. This helped ensure clarity of the questions and also provided an 
estimate of the time needed to complete the survey. Feedback was used to revise the survey questions as well as test 
the online survey tool. 

The first round survey included the principles determined during the 2009 study (York et al., 2009), in a 
randomly ordered list. The survey contained specific instructions as to how to access the survey, rate principles, and 
provide open-ended comments. The survey contained three parts: (a) a list of 59 principles to be rated on a 6-point 
Likert-scale (from 1 = strongly disagree to 6 = strongly agree), (b) a space for comments after each principle, and (c) 
a space for additional possible principles to be added by participants. Participants were asked to include comments 
to justify their ratings, to question or clarify the given principle, or to elaborate on the principle. 

Responses from Round I were analyzed and used to create the Round II survey. Panel ratings were 
analyzed using mean, median, mode, standard deviation, plus interquartile range (IQR). Frequency distributions and 
graphical representations were created for each principle. Principles that reached panel consensus in Round I were 
not included in Round II (Anderson-Woo, 2008). A principle reached consensus in Round I if either of the two 
following conditions were met: 

1. IQR less than/equal to 1 AND 75% agreement on a rating of 5 and 6 (agree, strongly agree) OR 1 and 2 

(disagree, strongly disagree). 

2. A 97% frequency rating in the 4, 5, 6 (mildly agree, agree, strongly agree) categories OR in the 1, 2, 3 

(mildly disagree, disagree, strongly disagree) categories (97% indicated all but 1 participant). 

The first condition allowed 29 principles to be removed and the second condition resulted in the removal of 1 
principle prior to the Round II survey. This means that the panel agreed that 30 of the original 59 represented 
instructional design principles. 

Any new principles suggested by participants were qualitatively analyzed in order to determine if that 
principle was new, or should be integrated into a previously existing principle (perhaps the participant used different 
wording but the meaning was the same). Before adding any new principles to the Round II survey, the research team 
reviewed the suggested principles, reworded some, and broke some into more than one. This resulted in 15 new 
possible principles for the Round II survey. Based on participants’ suggestions, six principles were reworded for 
Round II. 

Statistical measures (e.g., mean, median, mode, frequency, standard deviation) were included in Round II 
for each of the remaining 29 original principles. Panel comments from Round I were also included so participants 
could read other panel members’ justifications for their ratings. Participants also were presented with their original 



299 




responses to Round I and asked to either retain their original ratings or modify them based on the new information. 
Thus, in this round, participants rated 44 principles and provided comments to support their ratings. 

The results of the Round II survey were statistically analyzed within a week of the final submission to 
determine consensus. Those that reached consensus in Round II were not included in Round III. In Round II, the 
criteria for determining level of consensus were not as restrictive as Round I. That is, we decided to retain the first 
criterion but to lower the second criterion from 97% to 80% as this seemed more reasonable than the high level set 
for Round I. No literature was found regarding changing the level of consensus from one round to the next, only that 
level of consensus for any Delphi study is determined by the researcher (Hasson, Keeney, & McKenna, 2000; 
Powell, 2003). 

In addition, if a principle received a 20% or less stable rating from Round I to Round II, it was an 
indication that participants were not likely to change their ratings enough to come to consensus. To calculate 
stability, the frequency of Round I and Round II responses was determined. Following this, the net person-changes 
(total units of change/2) for a particular question was divided by number of participants (Scheibe, Skutsch, & 
Schofer, 1975, p. 279). Using a criterion of 20% or less stability from Round I to Round II, seven principles were 
eliminated before the Round III survey. 

In addition, two principles were determined not to be instructional design principles and were deleted 
before Round III. This was based on the fact that 80% or more of the participants disagreed with the principle (i.e., 
rated their level of agreement with the principles as 1, 2, or 3). This left 10 principles from Round II to be included 
in Round III. 

The Round III survey was created, reviewed, and disseminated to the panel within 1 week of receiving 
Round II submissions. Participants were again asked to submit their responses within 2 weeks. This final survey 
allowed the participants to review the feedback their fellow participants provided during Round II. As in Round II, 
participants were provided statistical measures such as mean, median, mode, frequency, and standard deviation for 
each of the 10 principles included in the survey. Similarly, participants were presented with their original responses 
and all panel comments from the previous survey and asked to retain or revise their original opinions based on the 
new information. 

The responses from the Round III survey were analyzed. Consensus and stability criteria remained the 
same as Round II. Analysis resulted in five additional instructional design principles, one statement determined not 
to be a principle, two statements that reached stability, and two statements that did not reach either stability or 
consensus. 



Results and Discussion 

This study was designed to identify the principles instructional designers use to organize their knowledge 
of the field, in general, and more specifically of the ID process. How do the principles designers use relate to the 
steps in the ID model and/or other “big ideas” (Bransford et ah, 2003) in the field? 

Three rounds of the Delphi resulted in panel consensus on 61/75 instructional design principles (see 
Appendix). Of these 61, 32 were identified as readily fitting into a general ID model (e.g., ADDIE). Thus, we 
organize the first section of our results and discussion to highlight how the identified principles address the five 
ADDIE components: Analysis, Design, Development, Implementation, and Evaluation. The second section 
discusses additional principles that, although they did not readily relate to the general ID model, related to other key 
concepts in the field, such as communication and project management. 

Analysis 



What principles do designers use when completing the analysis phase of the design process? Based on the 
responses of the Delphi participants, 10 principles were identified that relate to analysis tasks (see Table 1). These 
analysis principles centered around four main ideas: (1) determining if instruction is the solution to the problem, (2) 
examining the project constraints, (3) understanding the learner/audience and their prerequisite knowledge, and (4) 
determining the objectives or goals of the project. It also includes understanding the scope of the project and 
creating a statement of work with the client in order to reduce the possibility for miscommunication. 

Results reported by Rowland and DiVasto (2001) support our finding that analysis is one of the “big ideas’ 
that designers use when engaging in design work. Thel4 experts in the Rowland and DiVasto study all agreed that 
the instructional design “process includes thorough analysis, for example, of learners, task, and setting” (p. 14). In 
addition, three of their experts claimed that, in general, not enough analysis takes place. Their statement supports the 
principle, “Invest as much time as you can in your audience analysis.” 
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The principle, “Ask yourself, ‘Is instruction the solution to this problem?’” is also supported by Rowland 
and DiVasto (2001), whose instructional designers’ analyses “determined when instruction was the right answer” (p. 
15). Designers generally ask this question during the needs analysis phase. The ID literature also supports this 
principle (Romiszowski, 1981). Determining if instruction is necessary tends to be one of the first things a designer 
must do after communicating with the client in the first project meeting. 

Table 1 



Analysis Principles 



Component 


Principle 


ANALYSIS 


• 


Ask yourself, “Is instruction the solution to this problem?” 


• 


Problem 


• 


Invest as much time as you can in your audience analysis. 


• 


Need 


• 


Know your leamers/target audience. 


• 


Goals 


• 


Know your learners' prerequisite knowledge. 


• 


Learners 


• 


Needs analysis is the foundation for evaluation. 


• 


Context 


• 


Negotiate the scope of the project with the client and create a statement of 


• 


Constraints 




work upfront. 


• 


Content 


• 


Determine what it is you want your learners to perform after the instructional 


• 


Task 




experience. What is the criterion for successful performance? 


• 


Timeline 


• 


There are things that need to be determined at the front end in order to make 
you successful at the back end. 






• 


Constraints are a key to design. Look for constraints that have been placed on 
a project. 






• 


Never look at the problem at face value. You have to get to the core of the 
problem and solve all of the subproblems. 



Design 



What principles do designers use during the design phase of the ID process? Related to design, 1 7 
principles were identified by the panel of participants (see Table 2). This component included the largest number of 
principles identified by the panel. The design principles centered on the following three main ideas: (1) solving the 
problem, (2) identifying potential learner activities, and (3) considering the technology. 

Similar to the results of this study, experts in the Rowland and DiVasto study (2001) agreed that design was 
one of the most important features of an instructional design project. Project goals, outcomes, and/or deliverables are 
widely discussed in the instructional design literature (Rowland, 1993). 

The principle, “Generate multiple possible solutions that will solve the problem,” is supported by Liu, 
Gibby, Quiros, and Demps (2002) who suggested that instructional designers must use their best judgment in 
creating a solution for the client. Their discussion does not specifically mention developing multiple solutions, 
however because ill-structured problems typically have multiple solutions, the designer needs to decide which is the 
best to recommend to the client (Jonassen, 1997). 

The principle, “When designing instruction, consider active learning,” is supported by Mayer (2003), who 
recommended different methods to promote active learning, even when using non-interactive media. Participants 
also agreed that scaffolding was needed, but disagreed as to the timing and quantity of the scaffolding. The 
instructional design literature supports the concept of using scaffolding, but the use of it depends on the context 
(Van Merrienboer, Kirschner, & Kester, 2003). 



Table 2 

Design Principles 

Component 

DESIGN / SELECT 

• Objectives 

• Instructional 
Strategies 

• Visual design/ 
storyboards 

• Assessments 



Principle 

• When faced with something complex, look for previous examples that have 
characteristics you can draw upon, that can give you ideas on how to solve 
the problem. 

• Approach the design problem with the end in mind. What are the 
deliverables? What are the learning/performance outcomes? 

• Generate multiple possible solutions that will solve the problem. 

• When designing instruction, consider the context in which the learning will 
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• Media / Methods be applied. Ask yourself, "How can I put learning into context?” 

• When designing instruction, consider active learning. Ask yourself, “How 
can I make learners more actively engaged?” 

• Determine what will keep the learner motivated during the instructional 
experience. 

• Consider utilizing scaffolding in your instructional experience. Give the 
learner the tools they need to succeed. 

• Be sure the instruction gives learners the opportunity to make choices. 

• Ensure that design speaks to a value chain of learning, i.e., that learning 
contributes to behaviors and that behaviors contribute to organizational or 
business results. 

• Understand the learning associated with the technology. 

• Don’t let technology drive the design. 

• Resist the technical expert's propensity to focus on the most complex or 
innovative aspects of a product. Remember the novice learner who needs to 
build basic skills. 

• It is the instructional designer's job to press for quality in the design. 

• Resist the SME's (subject matter expert’s) desire to teach the solution to the 
hot problem of the day... unless it is a common problem seen by the average 
learner. 

• Prepare to do a lot of work that is never going to show up in the final 
product. 

• When designing instruction, think about Elaboration Theory. Ask yourself, 
“What’s the ‘big picture’ to which the components are attached?” 

• Design continues through the delivery or implementation phase. 



Development 

What principles do designers use when completing the development phase of the ID process? In the 
development component, three principles were identified (see Table 3). The three principles related to being part of 
the production process, dealing with the technology involved, and allowing the content, not the technology, to guide 
how users interact with the training. Similarly, Liu et al. (2002) described how designers needed to understand the 
project’s needs and determine which technologies could produce the best product for the learner. 



Table 3 

Development Components and Principles 



Component 



Principle 



DEVELOPMENT 

• Production of content 
and learning materials 



Allow the content to guide how users interact with the training (linear, user- 
driven, etc.) - not the tools used to develop the training. 

Technology can get in your way, and if you don’t deal with it you can get 
yourself into trouble. 

Make every effort to be part of the production process. 



Implementation 

The panel of participants in this study did not identify any principles that related directly to the 
implementation phase. This could be due to the fact that many designers tend to “pass” the instructional solution 
over to the client, who then implements it. Thus, the designers surveyed in this study may not have played a role in 
the implementation process and so the principles they used in practice did not fit into the implementation category. 
Furthermore, the Dick and Carey model does not contain a “box” for implementation (Dick, 1996), as 
implementation is not considered one of the main activities of an instructional designer (Wedman & Tessmer, 1993). 

An alternative explanation for the lack of implementation principles being identified in this study could be 
that the original 59 principles were derived from the stories of only 16 participants. Perhaps the second set of 
participants, in reading through the list, failed to consider additional principles, specifically related to 
implementation, that went beyond those on the initial list. 
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Evaluation 



What principles do designers use when evaluating the design solution? In this study, two principles were 
identified by the participants (see Table 4) that related to evaluation. These included conducting a pilot test and 
having both a SME and non-SME review the final project. Interestingly, Wedman and Tessmer (1993) found, based 
on the survey results of 73 designers, that pilot testing was the most-often omitted step of the instructional design 
process. There were several reasons for this omission such as time, money, and lack of support from the client. 
Keppell (2001) supported the principle that reviews with subject matter experts were a necessary component of the 
instructional design process. The content being designed could be quite unfamiliar to the designer, whereas the 
subject matter expert is highly knowledgeable about the subject. Keppell described an iterative process of 
explanation and clarification between the designer and the subject matter expert throughout the design process. 

Table 4 

Evaluation Components and Principles 

Component Principle 

EVALUATION • Always conduct a pilot. 

• Formative • When possible, have a subject matter expert AND a non-subject matter 

• Summative expert review the final product. 



In addition to the principles that related to the main components of an ID model, there were 29 additional 
principles identified by the Delphi panel. These principles clustered around three other components of design: (1) 
communication, (2) working with a client, and (3) project management, as well as those that related to specific 
characteristics of the designer and the design process (e.g., recognize your limitations, recognizing the uniqueness of 
each situation). 

Communication 

Communication is an important component in the instructional design process. The designer must have 
interpersonal skills such as being able to communicate with a number of people including the client, design team 
members, and other stakeholders, all of whom may use different terminology. McDonald (2008) explained, “Given 
the importance of communication in the work of instructional design, it is worthwhile to use the metaphor of 
instructional designers and their clients speaking different languages...” (p. 19). This component centered on two 
communication principles, “You need to understand and speak the language of your client,” and “Don’t use 
technical instructional design terminology with the client unless you have to.” Liu et al. (2002) suggested that 
instructional designers should either use the jargon of the client or everyday terminology when communicating. 
Miscommunication can be avoided by verifying information, asking questions, and by using visuals and other 
documents when communicating. 

Table 5 

Communication 

• Asa designer you need to listen more than you talk. 

• When verifying information, you often will learn more information. 

• Verify all the information you receive from the client to prevent miscommunication. 

• You need to understand and speak the language of your client. 

• Don’t use technical instructional design terminology with the client unless you have to. 

• Ask all possible relevant questions throughout the entire design process. 

• You are rarely going to collect all the desired outcomes with just one interview with the client. 

• When communicating with the client, use visuals and documents in order to prevent miscommunication. 



Client 



The relationship between the instructional designer and the client is an important one. Working with a 
client is one of the major responsibilities of an instructional designer (Liu et al., 2002; Rowland, 1993). There are a 
number of different principles noted by our participants related to the designer-client relationship. The client could 
be the subject matter expert (SME), owner, manager, or other stakeholder. It is the designer’s job to understand the 
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client, their culture, language, expectations, documentation, and communication. In addition, the designer needs to 
recognize the root of the problem, take prodigious notes, mock up a prototype, and make sure all stakeholders are 
updated throughout the process. 

This seems to imply that the designer needs to help the client understand how their stated expectations 
might translate into a design product (Liu et ah, 2002). Liu et al. (2002) stated, “Some clients need assistance in 
producing a clear definition of the problem they are trying to solve” (p. 205). The designer needs to explain the 
process of moving from a design to an actual product. As Liu et ah (2002) stated, “Clients may not be aware of the 
steps and tasks that a designer takes to get to the end product. Some clients expect the designer to start from scratch 
and create a polished product within a short time” (p. 205). 

Table 6 

Client 

• Be honest with the client. 

• You have to be sensitive to the context and the culture of the client. 

• You need to build trust with the client. This can be done through explaining what you are doing, why you are 
doing it, and how it is of value to them. 

• Figure out who all the stakeholders are in the room. And figure out who is not in the room that is still a 
stakeholder. 

• You need to manage the client’s expectations. 

• You have to determine if the client really knows what they want. 

• Subject matter expert’s documentation often fails to provide the necessary critical thinking. The SME forgets to 
tell you basic steps and concepts he forgot he once learned. 

• When multiple stakeholders are involved, ask your client to identify your "single point of contact" - make sure 
that person understands what is expected - gathering feedback on your design for you, getting approvals, etc. 

• Bring together the client and other stakeholders for synchronous meetings at each "gate" in a phased process. 

• The instructional designer should take prodigious notes during meetings. Do not rely purely on documentation 
or the SME. 

• Sometimes the client will not tell you all there is to know about a problem. 

• You may have to mock up something to show the client to make sure that you get all of the desired outcomes 
right. 

• The client thinks it is much easier to move from the conceptualization to the implementation than it actually is. 



Project Management 

Project management is a component of design that may or may not be the responsibility of the instructional 
designer. According to Liu et al. (2002), the instructional designer “must have design and project management 
skills” (p.197). As David Merrill stated, “you’re not going to go out and be designers even though we’re training 
you to be designers; you are going to go out and be managers” (personal communication, 2007). The one project 
management principle noted by practitioners was to involve the right people at the right time because design is a 
people process (as noted in Design Characteristics). 

Table 7 

Project Management 

• The team is critical. Involve the right people at the right time. 



Design Characteristics 

Finally, there are some general characteristics of design that designers embrace when working on a project. 
Some of these characteristics focus on the designer. For example, the designer needs to think abstractly yet 
acknowledge their own limitations. Nelson (1988) discussed that “the success of initial understanding and problem 
specification is directly related to the designer’s experience and knowledge in the problem domain” (p. 18). His 
statement indicates that if a designer accepts a job outside his/her area of expertise, he/she could have difficulties 
understanding and defining the design problem. 

The designer also needs to take into account foundational domain knowledge such as learning theories and 
different design models. Perez et al. (1995) noted that the novice instructional designers in their study were not able 
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to transfer “theoretical knowledge into practice” (p. 339), despite having completed foundational instructional 
design courses. Thus, experience appears to play a part in how designers utilize that foundational knowledge during 
the design process (Liu et al., 2002; Perez et al. 1995). 

Although every design situation is unique, if designers have previous experiences they can use as a starting 
point, they should do so. This has been widely corroborated in the instructional design literature (Ertmer et al., 2008; 
Hmelo-Silver et al., 2002; Liu et al., 2002; Nelson, 1988; Perez et al., 1995). Researchers agree that experienced 
instructional designers pull from their past experiences during the design process. The panel participants supported 
this principle with only three of them mildly disagreeing. 

Table 8 

Design Characteristics 

• Acknowledge your limitations. Don’t accept a job that is outside of your expertise. 

• You often don’t get to do the best instructional design you want due to constraints, resources, time, budget, etc. 

• Use previous experiences, if possible, as a starling point for new projects. 

• Be prepared to think abstractly. 

• Understand that every design situation is unique. 

• You need to know the theories. You need to know the models. You need to have that foundation. 

• Design is a people process. 



Summary and Implications 

Experts, in any field, have knowledge that is both well-organized and “conditionalized” (Bransford et al., 
2003). This means that experts can readily search their personal libraries to find information that is relevant to the 
particular situation (Kolodner, Owensby, & Guzdial, 2004). Bransford et al. provided an example of physics experts, 
who organized their mental models around the big ideas in the field (momentum, energy, etc.), and when asked to 
analyze a problem, went beyond the surface details to consider the underlying big idea. Similarly, Walker (1971) 
described how curriculum developers [designers] used a platform of ideas, comprising their beliefs and values, to 
determine the path to take during the design process. Dreyfus and Dreyfus (1986) also found that the perspective the 
expert brought to the task influenced their plan of attack. No matter the terminology used (big ideas, platform of 
ideas, frames of reference, etc.), designers are described as utilizing guiding principles during the design process. 

The overarching purpose of this study was to examine the principles that guided instructional designers’ 
practice and to identify the extent to which these principles incorporated components of the instructional design 
model. Based on the results of the Delphi process, we identified 61 principles that instructional designers used in 
their practice, of which 3 1 related to a general ID model (e.g., ADDIE). The other 29 principles related to three 
components of ID not typically described in models (communication, client, project management) as well as some 
general characteristics of design. 

In this study, more than half of the principles identified by participants related to components of the 
instructional design model, suggesting its continued importance to the practice of ID. Although the participants in 
this study did not refer to an ID model by name, and did not add a guiding principle to the Delphi list of principles 
that specifically mentioned using an ID model, many of the resulting principles related to the major tasks outlined by 
the model. Perhaps as Romisowski (1981) suggested, designers use the model as a heuristic, adapting it, as needed, 
to address the specific problem with which they are faced. Additionally, it is quite likely the 3 1 principles that 
related to the model would not exist in the experienced designers’ repertoire had they not been taught an ID model at 
some point in their education. Without having this foundation on which to build, their new knowledge and 
experiences may not have been as readily “categorized.” Similar to the concept of “indexing” in the case-based 
reasoning literature (Kolodner et al., 2004), the practitioners in this study may have indexed their past experiences 
using the categories outlined by a general ID model. As noted by Stepich and Ertmer (2009), the ADDIE model may 
serve as a useful index for practitioners given that it is easy to remember, easy to build upon, and part of the 
common language of design. 

In addition, 29 principles were identified that were not directly related to the instructional design model; 
instead they related to client interaction, communication, management, and other design characteristics. This 
suggests that the education of ID novices should incorporate concepts that go beyond the ID models. That is, ID 
education should be an amalgamation of both models and skills. Case-based learning is one method that has been 
used successfully to foster the necessary skills (e.g., communication, problem solving) within novice instructional 
designers. As Ertmer and Russell (1995) stated, using case studies in ID education provides “an effective means for 
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bridging the gap between theory’ and practice ” because it “ integrates content knowledge with strategic thinking ” (p. 
25). 

Previous research (Ertmer et al., 2008; Visscher-Voerman, 1999) has demonstrated that designers utilize 
“frames of reference” when solving design problems. These frames of reference are derived from the designer’s 
knowledge, experiences, and fields in which they work (Ertmer el ah). Although our participants did not mention 
frames of reference, explicitly, the guiding principles identified in this study were likely part of these frameworks. 
That is, the participants in our study delineated principles that were based on their prior knowledge and experiences. 
For example, one participant commented that he believed that experienced instructional designers had an 
“internalized theory” from which they approached new design projects. In many ways, this is similar to the 
definition of a frame of reference as described by Wilson (1997). Another participant described how her prior 
experiences are always influencing her work and decisions. 

If we equate principles with the core concepts or big ideas in the field, then the core concepts of design, 
with 17 principles, and analysis, with 10, as well as client interaction, with 13, may, perhaps, be most salient to 
designers’ practice. Of the 1 1 activities listed by Wedman and Tessmer (1993) as being common to the ID process, 
all but two comprise analysis and design components. The remaining two activities were evaluation activities. This 
supports the idea that analysis and design are two of the core concepts of ID. Client interaction was not included in 
Wedman and Tessmer’s survey because they only included components that were common to ID models. Of course, 
greater numbers of principles does not suggest greater importance. Rather, it may be possible that there are simply 
more elements contained in the design, analysis, and client interaction components of design. It would be necessary 
to verify this by surveying the participants and querying what they believe are the core concepts of ID. Again, this 
may differ depending on their design contexts and their current roles in the field. 

The results of this study have implications for the education of instructional designers. Because design is a 
problem-solving process, novices should understand what practicing instructional designers do and what principles 
they work from, rather than just memorizing steps in a model. Still, familiarity with the models appears to offer the 
basis for many of these principles. According to Stepich and Ertmer (2009) expert instructional designers “use their 
knowledge of ID models in flexible and dynamic ways” (p. 155). Even though novice instructional designers do not 
yet have the amount of experience that allows them to create a robust set of personal principles, their design 
performance has the potential to be improved if they know, a priori, some of the principles others use when solving 
instructional design problems. It is possible that novices could learn, vicariously, from the stories of experienced 
instructional designers including lessons learned and principles used (Jonassen & Hernandez-Serrano, 2002). Just as 
ID models are used to help novices understand the steps involved in the analysis, design, development, 
implementation, and evaluation phases of the process, principles might help them understand the overarching 
communication and management aspects, as well as other areas. For example, if novices are taught to verify all the 
information they receive from a client, they could improve their communication with the client. These results imply 
that ID principles should be incorporated within our ID programs, if they are not already. 

So, how can principles be incorporated into our ID programs, and more specifically, how can principles be 
used in conjunction with ID models? This could occur in a number of different ways as previously suggested: case- 
based learning, internships, or consulting. Vicarious learning from the stories (either narrative or video) of expert 
instructional designers could enhance the novice’s learning, particularly if the experts were asked to explicate the 
principles they applied during those problem-solving situations. A number of ID programs emphasize reflection at 
the end of lessons or internships. By including an explicit discussion of the learned instructional design principles, 
perhaps the learners can be reminded of principles they may otherwise have overlooked. Finally, ID instructors 
might consider integrating textbook information about the steps in the ID model with relevant principles identified 
in this study, as well as others found in the literature (e.g., Silber, 2007) 

The results of this study suggest that in addition to teaching novice instructional designers foundational 
models and theories, communication and project management skills should be taught. Communication skills could 
come in the form of English writing courses or practical experience such as internships and/or apprenticeships with 
real clients. In addition, although we only identified one project management principle, we should also consider 
teaching project management content. A good number of the principles identified by the panel imply that there 
should be a project management course in instructional design curricula, if it does not already exist. 

Limitations and Suggestions for Future Research 

This study was designed to generate a thoughtful analysis of principles used by experienced instructional 
designers. However, one limitation to this study was that the fields in which the instructional designers worked did 
not represent all possible areas of work for instructional designers; for example, designers working in the military 
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were not included because we did not have access to any. All judgments made by participants were based solely on 
their own knowledge, skills, and experiences and may not be representative of all instructional designers. The 
decision to start the first Delphi survey using the principles that emerged from the 2009 study also may have 
impacted the results. An open-ended question was included on the first survey to prevent the possible omission of a 
critical principle from the panel’s perspective; however starting with an initial list still may have unduly constrained 
the potential principles considered by the panel participants. 

This study is based on the assumption that we need to leam/teach more instructional design principles. 
However, it should be noted that more research needs to be conducted to determine if principles are already being 
taught and if not, how they might best be taught. Therefore, future research will focus on determining the best 
methods for sharing the resulting principles with novice designers and whether it impacts their experiences as 
instructional designers. Some questions we plan to pursue are: (a) Can we teach principles to novice instructional 
designers? (b) What methods should we use to provide this information (stories, cases, guest speakers)? and (c) How 
does this impact their practice? In addition, the relative importance of different principles to the instructional design 
process (as well as to instructional design field) needs to be examined. A more thorough examination of importance 
of the different principles should be undertaken. Perhaps a Delphi panel could rank order the current principles as to 
their importance to the instructional design process. It could also be productive to ask new instructional designers 
questions about their experiences such as: (a) What was the most important thing you learned, and (b) About what 
do you wish you learned more? 

A more thorough examination of the principles found in the current study and their inclusion in, or 
exclusion from, various other instructional design models as well as the textural descriptions of the models is needed 
(besides ADDIE). From that examination, there could arise a need for the creation of a new ID model that 
incorporates relevant principles not found in current ID models. 
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Instructional Design Principles 



Appendix 



Component 


Principle 


ANALYSIS 


1. 


Ask yourself, “Is instruction the solution to this problem?” 


• 


Problem 


2. 


Know your learners/target audience. 


• 


Need 


3. 


Know your learners' prerequisite knowledge. 


• 


Goals 


4. 


There are things that need to be determined at the front end in order to make 


• 


Learners 




you successful at the back end. 


• 


Context 


5. 


Negotiate the scope of the project with the client and create a statement of 


• 


Constraints 




work upfront. 


• 


Content 


6. 


Constraints are a key to design. Look for constraints that have been placed on 








a project. 




Timeline 


7. 


Needs analysis is the foundation for evaluation. 




8. 

9. 

10. 


Invest as much time as you can in your audience analysis. 

Never look at the problem at face value. You have to get to the core of the 
problem and solve all of the subproblems. 

Determine what it is you want your learners to perform after the instructional 
experience. What is the criterion for successful performance? 



DESIGN / SELECT 11. Generate multiple possible solutions that will solve the problem. 

• Objectives 12. When faced with something complex, look for previous examples that have 

• Instructional characteristics you can draw upon, that can give you ideas on how to solve 

Strategies the problem. 

• Visual design / 13. When designing instruction, consider the context in which the learning will 

storyboards he applied. Ask yourself, "How can I put learning into context?” 

• Assessments 14. When designing instruction, consider active learning. Ask yourself, “How 

• Media / Methods can 1 ma ke learners more actively engaged?” 

15. Determine what will keep the learner motivated during the instructional 
experience. 

16. Approach the design problem with the end in mind. What are the 
deliverables? What are the learning/performance outcomes? 

17. Consider utilizing scaffolding in your instructional experience. Give the 
learner the tools they need to succeed. 

18. Ensure that design speaks to a value chain of learning, i.e., that learning 
contributes to behaviors and that behaviors contribute to organizational or 
business results. 

19. Understand the learning associated with the technology. 

20. Don’t let technology drive the design. 

21. Resist the technical expert's propensity to focus on the most complex or 
innovative aspects of a product. Remember the novice learner who needs to 
build basic skills. 

22. It is the instructional designer's job to press for quality in the design. 

23. Be sure the instruction gives learners the opportunity to make choices. 

24. Resist the SME's (subject matter expert’s) desire to teach the solution to the 
hot problem of the day... unless it is a common problem seen by the average 
learner. 



25. Prepare to do a lot of work that is never going to show up in the final 
product. 

26. When designing instruction, think about Elaboration Theory. Ask yourself, 
“What’s the ‘big picture’ to which the components are attached?” 

27. Design continues through the delivery or implementation phase. 

DEVELOPMENT 28. Technology can get in your way, and if you don’t deal with it you can get 

• Production of content yourself into trouble. 

and learning materials 29. Allow the content to guide how users interact with the training (linear, user- 

driven, etc.) - not the tools used to develop the training. 

30. Make every effort to be part of the production process. 
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EVALUATION 31. Always conduct a pilot. 

• Formative 32. When possible, have a subject matter expert AND a non-subject matter 

• Summative expert review the final product. 



Characteristics of Design 
Communication 

33. Be honest with the client. 

34. As a designer you need to listen more than you talk. 

35. When verifying information, you often will learn more information. 

36. Verify all the information you receive from the client to prevent miscommunication. 

37. You need to understand and speak the language of your client. 

38. Don’t use technical instructional design terminology with the client unless you have to. 

39. Ask all possible relevant questions throughout the entire design process. 

40. You are rarely going to collect all the desired outcomes with just one interview with the client. 

4 1 . When communicating with the client, use visuals and documents in order to prevent miscommunication. 

Client 

42. You have to be sensitive to the context and the culture of the client. 

43. You need to build trust with the client. This can be done through explaining what you are doing, why you are 
doing it, and how it is of value to them. 

44. Figure out who all the stakeholders are in the room. And figure out who is not in the room that is still a 
stakeholder. 

45. You need to manage the client’s expectations. 

46. You have to determine if the client really knows what they want. 

47. Subject matter expert’s documentation often fails to provide the necessary critical thinking. The SME forgets to 
tell you basic steps and concepts he forgot he once learned. 

48. When multiple stakeholders are involved, ask your client to identify your "single point of contact" - make sure 
that person understands what is expected - gathering feedback on your design for you, getting approvals, etc. 

49. Bring together the client and other stakeholders for synchronous meetings at each "gate" in a phased process. 

50. The instructional designer should take prodigious notes during meetings. Do not rely purely on documentation 
or the SME. 

5 1 . Sometimes the client will not tell you all there is to know about a problem. 

52. You may have to mock up something to show the client to make sure that you get all of the desired outcomes 
right. 

53. The client thinks it is much easier to move from the conceptualization to the implementation than it actually is. 
Project Management 

54. The team is critical. Involve the right people at the right time. 

Other Design Characteristics 

55. Acknowledge your limitations. Don’t accept a job that is outside of your expertise. 

56. You often don’t get to do the best instructional design you want due to constraints, resources, time, budget, etc. 

57. Use previous experiences, if possible, as a starting point for new projects. 

58. Be prepared to think abstractly. 

59. Understand that every design situation is unique. 

60. You need to know the theories. You need to know the models. You need to have that foundation. 

61. Design is a people process. 
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